See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/43490394

Validation of frameworks of communicative competence and oral proficiency

assessment

Article
Source: OAI

CITATIONS READS

0 602

1 author:

Michael Haugh
The University of Queensland
122 PUBLICATIONS   1,871 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

corpus linguistics View project

Initial Interactions View project

All content following this page was uploaded by Michael Haugh on 06 May 2015.

The user has requested enhancement of the downloaded file.

VALIDATION OF FRAMEWORKS OF COMMUNICATIVE COMPETENCE
AND ORAL PROFICIENCY ASSESSMENT

Michael Haugh
The University of Queensland
1. INTRODUCTION

The paradigm of communicative competence 1 has had a marked influence on both second language
pedagogy and assessment. However, in spite of its theoretical importance, various frameworks of
communicative competence have received little empirical validation. In this paper native speaker
evaluations of non-native speakers of Japanese are analysed by factor analysis in order to
investigate the underlying dimensions of the native speaker oral proficiency construct, since the
degree of consistency between the dimensions underlying native speaker evaluations of non-native
speakers and frameworks of communicative competence is indicative of the validity of current
frameworks for language assessment. In addition, as Clark and Lett (1988) point out:
"...(any) scale purporting to define and quantify second language proficiency...should take
adequately into account native-speaker judgements of the nature and effectiveness of the
examinee’s communication attempts, because it is precisely with native interlocutors that the
examinee would be expected to communicate in the real-life contexts that the scale is intended to
reflect...”
(Clark and Lett 1988: 57-58)
A review of the current state of communicative competence theory and validation studies is
followed by a summary of the main findings of this research.

2. THE COMMUNICATIVE COMPETENCE PARADIGM

The term communicative competence, first coined by Hymes (1967, 1972), grew out of Chomsky’s
(1965) distinction between competence and performance. The basic definition of communicative
competence is relatively straightforward, and has changed little since it was originally postulated by
Hymes:
“...the ability not only to apply the grammatical rules of a language in order to form
grammatically correct sentences but also to know when and where to use these sentences and to
whom...”
(Richards et al 1992: 65)
The way in which this concept is delineated, has received considerable attention over the past thirty
years, and although the most recent frameworks are more sophisticated and involve more factors
than previously included, the two basic aims of building communication competence frameworks,
to provide a valid basis for (oral) proficiency assessment and to function as a conceptual basis for
second language pedagogy, have not changed.
The framework for communicative competence (or communicative language ability as it is termed
by Bachman 1990a) as proposed by Bachman and Palmer (1996) is the most recent significant re-
working of Canale and Swain’s framework which dominated research into oral proficiency
assessment during the 1980s and early 1990s 2. Firstly, Bachman and Palmer hypothesized that
strategic competence is independent of language competence, and it is through the mechanisms of
strategic competence that language and topical knowledge, influenced by affective schema and
personal characteristics, are utilized in communication:
“...(Strategic competence is a) set of metacognitive components, or strategies, which can be
thought of as higher order executive processes that provide a cognitive management function in
language use, as well as in other cognitive activities...”
(Bachman and Palmer 1996: 70)
Secondly, they included personal characteristics (such as age, sex, L1, educational background et
cetera), topical knowledge (structures in long-term memory), and affective schemata (which are

1
affective or emotional correlates of topical knowledge) as factors in the communicative competence
framework. Finally, they reorganized the remaining three competencies postulated by Canale and
Swain into two domains of language knowledge, the second of which contains a newly postulated
functional (or illocutionary) competence. The first of these is organizational knowledge. This
includes grammatical knowledge (or competence) and textual competence, which are equivalent to
Canale and Swain’s grammatical and discourse competencies respectively. These influence the
organization of utterances/sentences and text. The second is pragmatic knowledge which describes
how utterances/sentences are related to the communicative goals of the users of the language, and
includes functional (or illocutionary) competence and sociolinguistic competence (similar to the
component in Canale and Swain’s model although more detailed 3). Functional competence enables
the interpretation and creation of relationships between utterances and texts, and the intentions of
the language user. Bachman and Palmer classify all ‘functions’ into four macro-functions:
ideational (express meaning and exchange knowledge), manipulative (affect the world around one),
heuristic (extend one’s knowledge of the world), and imaginative (the language itself is used for
humourous or aesthetic purposes). The framework of communicative competence (and terminology
associated) postulated by Bachman and Palmer, in particular their two domains of knowledge
(organizational and pragmatic knowledge), is used as the basis of discussion in this study.

3. VALIDATION OF FRAMEWORKS OF COMMUNICATIVE COMPETENCE

Essentially two different approaches have been employed to validate frameworks of communicative
competence. Descriptive validation involves researchers discussing the merits of various
frameworks qualitatively, whereas empirical validation involves statistical analyses of rater
evaluations of testees’ oral proficiency 4 . It is also important to note whether the framework
concentrates on inputs (factors involved in producing or interpreting utterances or texts) or on
outputs (criteria upon which communicative competence is judged). Although inputs such as
language knowledge or strategic competence can also be outputs, outputs (such as
comprehensibility or fluency) cannot always be regarded as inputs. Frameworks of communicative
competence and oral proficiency can be classified and related using these two parameters:
- Descriptive-output frameworks 5
- Descriptive-input frameworks
- Working-output models 6
- Working-input models
Significant models and studies are classified in Figure One below:

Descriptive

Canale and Swain (1980)

ACTFL (1986)
Bachman and Palmer (1996)

Input Output

Chalhoub-Deville (1995)

Ideal? Hadden (1991)

Harley et al (1990)

Bachman and Palmer (1982)

2
 Working

Figure One: A Framework for analysing models of communicative competence and oral
proficiency

Working-input models of communicative competence are argued to be the ideal conceptualisation

in this paper, as they are empirically validated and based upon inputs which can be both assessed
and taught (as opposed to some output criteria). Descriptive-input framworks of communicative
competence can be used in studies which attempt to build working-output models of communicative
competence (such as Bachman and Palmer 1982 or Harley et al 1990). The aim of these studies is
to infer a working-input model of communicative competence which would be a psychologically
sound representation of communicative competence. Therefore in order to successfully construct a
working-input model of communicative competence, both descriptive validation 7 (of descriptive-
input frameworks) and empirical validation (of working-output models) needs to be undertaken.

3.1 Descriptive validation of Bachman and Palmer`s descriptive-input framework

A review of the literature suggests that a number of limitations in Bachman and Palmer`s
framework of communicative competence need to be acknowledged. Firstly, the conceptualisation
of sociolinguistic competence is not well defined. Loveday’s (1982) distinction between
appropriate forms (symbolizing patterns) and the underlying principles and conventions of
conversation (framing patterns), is not duly acknowledged. The work on sociosemantics 8 (for
example, Halliday 1973, 1978) is also largely ignored, and the role that lexicalized sentence stems 9
(Nattinger and DeCarrico 1992, Pawley and Syder 1983) play in the ‘naturalness’ of native speaker
speech is not formalized in Bachman and Palmer’s framework. The use of the vague descriptor
‘naturalness’ in Bachman and Palmer’s sociolinguistic competence is indicative of the further work
that needs to be done in this area. Secondly, there needs to be more investigation into the
interaction between components of communicative competence in the manner of Halliday’s
model 10 (1978). As Savignon (1983) points out, it seems more logical to build frameworks of
communicative competence which are interactive as it is difficult to consider one component of
communicative competence without reference to others. For example, consider the situation where
a student who is a second language learner of Japanese says to her teacher, “Ashita sensei ga
kimasu ka?” (will you (the teacher) come tomorrow – polite level) instead of “Ashita sensei ga
koremasuka?” (super-polite level). In this case it is difficult to judge whether this is a
sociolinguistic competence error 11 (that is, the student is unable to utilise appropriate politeness
levels according to the characteristics of the hearer) or a grammatical competence error (that is, the
student simply does not know the super-polite form of `to come`). In other words, the realisation of
sociolinguistic competence may be dependent in some cases upon a sufficient level of grammatical
competence, as this example demonstrates. Thirdly, apart from the work of Higgs and Clifford
(1982), there have been no significant attempts to estimate the relative contributions of the different
components to the overall level of communicative competence, which is clearly one of the keys to a
more complete understanding of the communicative competence construct. Another weakness
pointed out by McNamara (1995) is that functional (or illocutionary) competence is more than just
knowledge and it overlaps with strategic competence. Although the introduction of functional
competence into the communicative competence framework is useful in that it increases the role
context plays in communicative competence theory, it is doubtful whether it can be classified as a
component separate from strategic competence. Finally, a paralinguistic competence factor needs
to be considered as part of communicative competence, as paralinguistic features are an essential
component of communication (Loveday 1982, Makino 1991).
The first two issues (sociolinguistic competence and a interactional-component framework of
communicative competence) are considered further in the empirical validation section of this study.
The issue of the relative contributions of different components of communicative competence is

3
considered further in Haugh (1998b). The final two issues (functional and paralinguistic
competence) were beyond the scope of this study, but are areas in need of further investigation.

3.2 Empirical validation of frameworks of communicative competence

There have been essentially two approaches to empirical validation of frameworks. One approach
has focused on the number of testees using factor analyses of multi-trait, multi-method matrices
(testee-oriented empirical validation). The second approach has concentrated upon the raters
(testers) using statistical analysis of the underlying dimensions of native speaker evaluations of
non-native speakers (rater-oriented empirical validation). The second approach is the one utilised
in this study, but due to the close relationship between the two approaches, testee-oriented
empirical validation studies are also discussed.

3.2.1 Testee-oriented empirical validation studies

One of the major assumptions underlying both Canale and Swain’s, and more recently Bachman
and Palmer’s framework, is that communicative competence is modular (that is, not dependent on
one single psychometrically isolatable factor, but involving a number of distinct ‘abilities’). A
number of studies have investigated this assumption, and although it has been clearly shown that
Oller’s unitary factor hypothesis (Oller 1979) is flawed (Carroll 1983, Farhady 1983, Purcell 1983,
Upshur and Homburg 1983), there have been no conclusive empirical studies supporting one
particular framework of communicative competence. Research by Bachman and Palmer (1982),
Hinofotis (1983), Raffaldini (1988), Fouly et al (1990), and Harley et al (1990) all utilized
confirmatory factor analyses on multi-trait, multi-method matrices, in order to attempt to validate
various components of proposed frameworks of communicative competence. Bachman and Palmer
(1982) found that the best fitting model was one consisting of a general factor plus two specific trait
factors (grammatical/textual competence and sociolinguistic competence), but this model was not
strongly supported. Hinofotis (1983) did not successfully validate components within language
knowledge, but found support for three components - language proficiency, delivery, and
communication of information. Raffaldini (1988) found that grammatical and sociolinguistic
competence are unrelated using factor analysis and orthogonal (varimax) rotation 12. Fouly et al
(1990) found support for the modular communicative competence hypothesis, but could not
determine whether the separate traits were all correlated with each other, or were influenced by a
single higher-order factor (that is, general factor plus specific trait factors model). Finally Harley et
al (1990), in a very extensive study, failed to find any support in their factor analysis results for
Canale and Swain’s framework of communicative competence. Moreover, the use of Canale and
Swain’s framework as the basis for confirmatory factor analysis was criticized by Schachter (1990)
due to the conceptual overlap of some of its components. (For example, she claims that there is no
justification for separating discourse (or textual) and sociolinguistic competencies). Bachman
(1990b) also pointed out that the use of orthogonal rotation, as opposed to oblique rotation, was
flawed as one would not expect the separate components of communicative competence to be
entirely uncorrelated, although according to Hinofotis (1983) using different rotation methods
makes no real difference in the final analysis. The difficulties in operationalizing the constructs
involved in communicative competence and the complexity of the statistical analyses involved has
hindered much of the research in this area. However, in spite of the continuing debate about the
most appropriate methodology, it appears from testee-oriented empirical validation that
communicative competence is modular, consisting of at least two components: grammatical
competence and sociolinguistic competence.

3.2.2 Rater-oriented empirical validation studies

4
A number of studies of native speaker evaluation of non-native speech have investigated the factors
underlying the oral proficiency/communicative competence construct using exploratory factor
analysis and other related statistical techniques. It is clear from both Chalhoub-Deville (1995,
1996) and Hadden (1991) that the background of the native speakers (for example, teachers versus
non-teachers) affects the perception of this construct. In addition, Chalhoub-Deville demonstrated
that the nature of the oral proficiency construct depends upon the text evaluated. Nevertheless
some similarities can be found amongst these three studies in terms of the dimensions extracted.
Albrechtsen et al (1980) extracted four dimensions, including language, personality,
comprehension, and a constant factor. The five dimensions found by Hadden (1991) were linguistic
ability, personality, comprehensibility, body language, and social acceptability, although only four
(linguistic ability and comprehensibility combined) were found to underlie the evaluations of non-
native speakers by non-teachers. Chalhoub-Deville (1995, 1996) extracted only three dimensions,
grammar-pronunciation, creativity in presenting in information, and amount of detail. It should be
noted that the number and kind of dimensions extracted are dependent on what items (for example,
personality factors, grammar factor, fluency et cetera) are presented to be evaluated by the native
speakers, and these differences resulted in the different number and nature of dimensions extracted
by these three studies.

3.2.3 Objectives of the study

The primary aim of this study was to use a investigation of the dimensions underlying Japanese
native speakers` evaluations of non-native speakers of Japanese to explore the validity of the two
domains of knowledge found in Bachman and Palmer`s framework of communicative competence.
The methodology employed was similar to previous rater-oriented empirical validation studies.

4. METHODOLOGY

Two phases were involved in this research:

1. Collection of samples of non-native speaker interlanguage (primary data).
2. Collection of native speaker evaluations of that interlanguage (secondary data).
The primary data was collected by means of dyadic interviews (between a non-native speaker and a
native speaker) conducted in Japanese, and the primary consideration was validity, in particular
contextualisation and authenticity. Audio-tapes, rather than video-tapes, were made of the
interviews in order to neutralize the possible influence of the physical appearance of the non-native
speakers upon the evaluations by the native speakers. Four intermediate to advanced non-native
speakers of Japanese 13, who are all graduates of the University of Auckland were selected by the
researcher. The four speakers 14, all native speakers of English, were chosen in order to have a
range of intermediate through to advanced level speakers of Japanese, and also to have a balance in
gender. Subjects One and Two were teachers of Japanese, and Subjects Three and Four were
students at the University of Auckland. Their ages ranged from twenty two to thirty, and the length
of their formal instruction of Japanese varied from three to nine years. All four subjects had lived
in Japan for at least a total of two years. They completed both an informal interview (with a
Japanese native speaker in her twenties on the topic of the subjects experiences in Japan) and a
formal interview (with a Japanese native speaker in her forties on the topic of education systems in
New Zealand and Japan). One and a half minute extracts from the two kinds of interviews were
chosen using non-linguistic criteria to reduce researcher bias 15.
The secondary data was obtained using a specifically designed evaluation sheet written in Japanese
which involved subjective rating scales of linguistic criteria. The linguistic items were designed to
account for the separate components of communicative competence as proposed by Bachman and
Palmer (1996) (items one to twelve and item seventeen), and for the more global linguistic criteria
utilized in ACTFL guidelines (ACTFL 1986, 1987). Thus the native speakers were asked to
evaluate the non-native speakers from both a ‘structuralist’ point of view and also from a ‘global’

5
perspective. A Likert-like linear scale (a seven-point scale marked with ‘very problematic’ at one
end and ‘excellent’ at the other end) was utilized so that scaled data with interval properties would
be generated. Fifteen native speaker respondents were asked to listen to three of the non-native
speakers’ speech samples from both the formal and informal interviews and then score the subjects
using the evaluation sheet while listening to the speech samples twice more. Ten of the respondents
were female, and eleven were between twenty and twenty nine (the remaining four were between
thirty and thirty nine). The region of Japan from which the respondents came were as follows:
Chugoku = 2, Kanto = 5, Kinki = 4, Kyushu = 4. All fifteen respondents could speak English
(mainly at an elementary level), and ten had received tertiary education. Six of the respondents
were students, with the other nine working in the tertiary sector (for example, company workers or
professionals).
All the subjects (non-native speakers) and respondents (native speakers) were of approximately the
same generation - people in the age group 20-35 - and thus this study investigated the evaluation of
young adult non-native speakers by their peers.

4.1 Data analysis

Exploratory factor analysis was performed using SAS version 6 (maximum likelihood extraction).
The number of factors to be selected for oblique transformation (promax) was determined by
inspection of the initial extraction of factors. Oblique transformation assumes that the underlying
factors will be correlated as one would expect for the communicative competence construct. Five
factors were selected as they appeared to best account for the data before transformation (the
eigenvalues were 0.5 or greater). The cut-off level for factor loadings was taken as 0.40 for the
oblique transformed data and was justified due to the limited sample size (n=15, number of
evaluations = 45) and large number of items (twenty two). In the case of this factor analysis this
cut-off point is definitely not a rigorous reflection of statistical significance, but is used purely in a
descriptive sense (Cliff 1987: 338).

4.2 Reliability and Validity

There was a high level of intra-rater reliability (Spearmans’ rho = 0.83) and an acceptable level of
inter-rater reliability (Pearson’s product moment correlation coefficient = 0.71) estimated for this
survey. The intra-rater reliability was calculated by comparing the overall scores assigned by the
respondents and the mean for the twenty two criteria, and the inter-rater reliability was calculated
according to Hatch and Lazaraton (1991: 533-534). The level of inter-rater reliability in particular
indicates that a large sample size was not of paramount importance (as the less variation that there
is between individuals, the smaller the sample that is required to accurately estimate that variation).
The low level of test-retest reliability (two respondents were asked to repeat the evaluation, and a ρ-
value of 0.56 was calculated) was most likely a result of individual variation rather than a trend, as
the level of intra-rater reliability was found to be so high, but this would require further
investigation to confirm.
The validity of this survey is argued to be high, because relatively authentic interlanguage samples
were collected. Research into interlanguage suggests that vernacular conversation obtained
surreptitiously would be the most authentic and representative sample of non-native speech.
However, in order to standardize and thus allow comparisons between different non-native
speakers, interviews were utilized to obtain interlanguage samples. It is argued by the researcher
that the samples are authentic examples of interview discourse (which is related to conversation),
that the use of both formal and informal interviews allowed for both ‘monitored’ and more
vernacular-like interlanguage to be elicited, and that this combined with the contextualisation of the
interlanguage inferred a sufficiently high level of validity upon these samples. Moreover, the

6
triangulation of native speaker evaluations with interviews and the analysis of the transcripts by the
researcher indicated a high level of consistency between the three different means of evaluating the
non-native speakers 16. The quality of recording was not significantly correlated with the overall
score (r2 = 0.10, p > 0.05), and did not affect evaluations of pronunciation or accent either (r2 =
0.07, 0.06 respectively, p > 0.05) and thus did not affect the results. Although a small sample size
was used, the high level of reliability and validity indicate that the conclusions of this study warrant
serious consideration.

5. RESULTS AND DISCUSSION

Factor analysis and oblique transformation of the data yielded five correlated factors. Factor
loadings greater than 0.40 were considered to be significant in this interpretation and are boxed in
Table One. h2 is the common factor variance, which represents the proportion of variance of that
particular variable which is accounted for by the five factors. Eigenvalues are mathematical
coefficients and were used to calculate the ‘% Variance’, which is the percentage of the total
variance of the five factors explained by each factor.
Item I II III IV V h2
Accuracy of grammar (3) .72 .17 .07 .05 -.06 .79
Accent (2) .70 .14 .03 -.11 .16 .77
Pronunciation (1) .62 .44 -.07 -.10 .08 .86
Range of grammar (4) .60 .18 -.02 .09 .09 .69
Expressiveness (14) .59 -.21 .37 .00 .17 .70
Joining sentences (7) .56 .34 -.09 .09 .22 .93
Communication strategies (13) .50 .04 .31 .19 -.00 .78
Appropriate vocabulary (6) .44 .25 .17 .30 -.20 .75
Discourse structure (8) .47 .10 .22 .35 -.06 .83
Range of vocabulary (5) .11 .81 -.08 .10 .01 .80
Accuracy (19) .20 .63 .13 .14 -.06 .85
Comprehensibility (16) .09 .57 .34 -.05 .17 .92
Native-like (10) .21 .51 .21 -.02 .17 .84
Complexity (20) .20 .46 .16 .12 .17 .82
Fluency (15) .20 .44 .44 -.17 .12 .87
Speed (18) .02 -.01 .89 -.01 .12 .90
Confidence (22) .21 .08 .65 .07 .01 .82
Express feelings/knowledge (17) -.16 .34 .53 .42 -.15 .86
Register (9) .00 -.01 -.07 .86 .29 .89
Appropriateness (21) .01 .01 .05 .86 .01 .81
Range of cultural references (12) .07 -.08 .06 .16 .88 .94
Appropriateness of cultural references (11) .03 .22 .03 -.01 .78 .88
Eigenvalues 14.91 1.32 0.94 0.62 0.49
% Variance 82 7 5 3 3

Table One: Oblique transformed factor matrix for native speaker evaluation of non-native
speakers

Five factors were found to underlie native speaker evaluations by factor analysis. The two factors
which accounted for most of the variance of the data set, called Grammatical-discourse factor
(factor one) and Language factor I (factor two), contained components of organizational knowledge
(grammatical and discourse competencies) and global criteria. The other three, called Fluency
factor (factor three), Appropriateness factor (factor four) and Japanese culture factor (factor five),
were smaller and more clearly delineated.

7
The Grammatical-discourse factor contained items mainly related to grammatical and discourse
competencies, but also contained the global criteria of expressiveness and communication
strategies. The relationship of the global criteria to the other items was rather indirect, but
expressiveness was most likely related to accent and pronunciation, and communication strategies
were most likely linked to grammar and vocabulary (that is, to be used when suitable grammar or
vocabulary was not known by the non-native speaker). The grammatical-discourse factor overlaps
conceptually with the organizational knowledge component of Bachman and Palmer’s (1996)
framework of communicative competence.
Language factor I contained a variety of global criteria plus two components of grammatical
competence. The relationship of the two specific criteria to the global criteria was difficult to
ascertain, and so the factor could only be characterised as a general language factor. The cluster of
items suggests that range of vocabulary and pronunciation were strongly related to
comprehensibility, fluency and native-like speech. This indicates that these specific criteria may be
important in native speaker evaluations, as those global criteria are definitely important in native
speaker evaluations (Haugh 1998b).
The Fluency factor contained items related to producing fluent speech in order to express thoughts
and feelings, and confirmed the relationship between fluency, speed and confidence.
The Appropriateness factor contained two items related to producing appropriate speech, depending
on the situation, and these were related to the ability to express thoughts and feelings.
The Japanese culture factor contained two items relating to range and appropriate use of Japanese
cultural references and expressions. The existence of two factors related to sociolinguistic
competence suggests that the sociolinguistic competence construct may require further re-working
before it becomes a psycholinguistically sound representation within the communicative
competence construct.
The five factors extracted from this data do not match factors obtained from previous studies
(Albrechtsen et al 1980, Chaulhoub-Deville 1995 and Hadden 1991). However, the number and
type of dimensions that will be extracted by factor analysis in this kind of study depend to a large
extent upon the type of text evaluated (for example, interviews versus narration), and the number
and nature of the criteria evaluated by the native speakers. The text used in this survey was similar
to the other three studies, but a larger number of criteria were evaluated in order to obtain a more
comprehensive view of native speaker evaluation of linguistic criteria. The different results are
thus not surprising, as previous studies have not emphasised linguistic criteria to such a large extent
as in this study. The difficulties in comparing studies illustrates the need to use more standardised
methodology, although to decide this would not be an easy task.

6. CONCLUSION

The factors obtained were not inconsistent with the organisational knowledge and the
sociolinguistic competence components of communicative competence as postulated by Bachman
and Palmer (1996), or with the studies by Bachman and Palmer (1982) and Raffaldini (1988).
However, the fact that the factors extracted were not well delineated along the lines of Bachman
and Palmer`s postulated domains of knowledge suggests that an interactive model of
communicative competence, which describes the overlap and interactions of the components of
communicative competence, may be more valid 17. The compartmentalisation of the two domains of
knowledge of communicative competence is also problematic when one considers that the
realisation of one component may be dependent on a certain level of ability in another component
as discussed previously. However, this tentative conclusion is tempered by the fact that the lack of
clear delineation of these components may simply be a function of the lack of clear definition of the
communicative construct in the minds of the native speaker subjects. In addition, the results of this
survey need to be considered with considerable caution as the number of evaluations is too small to
give definitive proof, and in any case, factor analysis should be regarded as an exploratory tool
only, due to the controversial nature of its mathematical foundations (Cliff 1987).

8
A review of descriptive validation studies and the results of this survey also suggest that the
sociolinguistic competence component, while clearly a macro-component of communicative
competence, is in need of some reworking if it is to be a valid representation of the concept of
appropriateness. The sociolinguistic component may involve more than one sub-component
including ‘socio-cultural knowledge’ (for example, knowledge of Japanese festivals or lexicalised
sentence stems) and knowledge of sociolinguistic principles (for example, how to show social
distance depending on whom one is speaking to, or what topic of conversation is more appropriate
for various age groups). The application of sociolinguistic principles, as opposed to grammatical
rules, is governed primarily by the probability of occurrence of a finite number of conventions (for
example, Brown and Levinson’s (1987) claim that negative politeness strategies are emphasised
when conversing with an older stranger). This proposed division is supported by the Japanese
culture factor and Appropriateness factor which were extracted in this survey.
The results of this study suggest that developing a more interactive framework of communicative
competence could be a worthwhile avenue of research to pursue, and that the sociolinguistic
component may consist of more than one sub-component, including socio-cultural knowledge and
sociolinguistic principles. However, to give more credence to these preliminary conclusions further
research with larger sample groups, with different backgrounds, and with different test methods 18
(for example, role play or narration) needs to be undertaken. The implication of this study for
researchers trying to develop a working-input model of communicative competence is that the use
of native speakers who have no specific training in linguistics is both possible and desirable. Thus,
further rater-oriented empirical validation studies can form an integral part of future communicative
competence and oral proficiency research.

ACKNOWLEDGEMENTS

I would like to thank my supervisors Jack Richards and Thomas Leims for their invaluable advice
and support while I was conducting the research for this paper. I would also like to thank the staff
in both the Institute of Language Teaching and Learning (in particular Breon Gravatt and Marilyn
Lewis) and the Department of Asian Languages and Literatures (especially Mari Imae and Chako
Eastwick-Field), at the University of Auckland.

REFERENCES

Albrectsen, D., B. Henriksen, and C. Faerch (1980). ‘Native speaker reactions to learners’ spoken
interlanguage’. Language Learning 30, 2: 365-396.
American Council on the Teaching of Foreign Languages (ACTFL) (1986). ACTFL Proficiency
Guidelines. New York: ACTFL.
ACTFL (1987). ACTFL Japanese Proficiency Guidelines. New York: ACTFL.
Bachman, L. (1990a). Fundamental Considerations in Language Testing. Oxford: Oxford
University Press.
Bachman, L. (1990b). ‘Constructing measures and measuring constructs’. In, B. Harley, J.
Cummins, M. Swain, and P. Allen (eds.): 26-38.
Bachman, L. and A. Palmer (1982). ‘The construct validation of some components of
communicative proficiency’. TESOL Quarterly 16, 4: 449-465.
Bachman, L. and A. Palmer (1996). Language Testing in Practice. Oxford: Oxford University
Press.
Bayard, D. (1990). ‘God help us if we all sound like this: attitudes to New Zealand and other
English accents’. In A. Bell and J. Holmes (eds.): 67-96.
Bell, A. and J. Holmes (1990). New Zealand Ways of Speaking English. Wellington: Victoria
University Press.
Brown, G., K. Malmkjaer and J. Williams (1996). Performance and Competence in Second
Language Acquisition. Cambridge: Cambridge University Press.

9
Brown, P. and S. Levinson (1978). Politeness. Some Universals in Language Usage. Cambridge:
Cambridge University Press.
Canale, M. (1983). ‘From communicative competence to communicative language pedagogy’. In,
J. Richards and R. Schmidt (eds.): 2-27.
Canale, M. (1988). ‘The measurement of communicative competence’. Annual Review of Applied
Linguistics 8: 67-84.
Canale, M. and M. Swain (1980). ‘Theoretical bases of communicative approaches to second
language teaching and testing’. Applied Linguistics 1, 1: 1-47.
Canale, M. and M. Swain (1981). ‘A theoretical framework for communicative competence’. In,
A. Palmer, P. Groot, and G. Trosper (eds.), The Construct Validation of Tests of
Communicative Competence. Washington, DC: TESOL: 31-36.
Candlin, C. (1986). ‘Explaining communicative competence limits of testability?’. In, C.
Stansfield (ed.), Towards Communicative Competence Testing: Proceedings of the Second
TOEFL Invitational Conference. Princeton, NJ: Educational Testing Service: 38-57.
Carroll, J. (1983). ‘Psychometric theory and language testing’. In, J. Oller Jr. (ed.): 80-107.
Chaulhoub-Deville, M. (1995). ‘Deriving oral assessment scales across different tests and rater
groups’. Language Testing 12, 1: 16-33.
Chaulhoub-Deville, M. (1996). ‘Performance assessment and the components of the oral construct
across different tests and rater groups’. In M. Milanovic and N. Saville (eds.), Performance,
Testing, Cognition and Assessment. Cambridge: Cambridge University Press: 55-73.
Chomsky, N. (1965). Aspects of the Theory of Syntax. Cambridge, Mass: MIT Press.
Clark J. and J. Lett (1988). ‘A research agenda’. In, P. Lowe Jr. and C. Stansfield (eds.), Second
Language Proficiency Assessment: Current Issues. NJ: Prentice Hall: 53-82.
Cliff, N. (1987). Analyzing Multivariate Data. New York: Harcourt Brace Jovanovich.
Cziko, G. (1984). ‘Some problems with empirically-based models of communicative competence’.
Applied Linguistics 5, 1: 23-38.
Farhady, H. (1983). ‘On the plausibility of the unitary language proficiency factor’. In, J. Oller Jr.
(ed.); 11-28.
Fouly, K., L. Bachman, and G. Cziko (1990). ‘The divisibility of language competence: a
confirmatory approach’. Language Learning 40, 1: 1-21.
Gorden, E. and M. Abell (1990). ‘This objectionable colonial dialect: historical and contemporary
attitudes to New Zealand speech’. In, A. Bell and J. Holmes (eds.): 21-48.
Hadden, B. (1991). ‘Teacher and non-teacher perceptions of second-language communication’.
Language Learning 41, 1: 1-24.
Halliday, M. (1973). Explorations in the Functions of Language. London: Edward Arnold.
Halliday, M. (1978). Language as a Social Semiotic. London: Edward Arnold.
Harley, B., J. Cummins, M. Swain, and P. Allen (1990). The Development of Second Language
Proficiency. NY: Cambridge University Press.
Hatch, E. and A. Lazaraton (1991). The Research Manual. Design and Statistics for Applied
Linguistics. Boston, Mass.: Heinle and Heinle.
Haugh, M. (1998a). The Reaction of Japanese to Non-native Speakers of Japanese. Unpublished
MA thesis, The University of Auckland.
Haugh, M. (1998b). Native Speakers’ Evaluations of Non-native Speakers of Japanese.
Unpublished manuscript.
Higgs, T. and R. Clifford (1982). ‘The push toward communication’. In, T. Higgs (ed.),
Curriculum, Competence, and the Foreign Language Teacher. ACTFL Foreign Language
Education Series Vol. 13. Lincolnwood, IL: National Textbook Company: 57-79.
Hinofotis, F. (1983). ‘The structure of oral communication in an educational environment: a
comparison of factor-analytic rotational procedures’. In, J. Oller Jr. (ed.): 170-187.
Hymes, D. (1967). ‘Models of the interaction of language and social setting’. Journal of Social
Issues 23, 2: 8-38.

10
Hymes, D. (1972). ‘On communicative competence’. In, J. Pride and J. Holmes (eds.),
Sociolinguistics. England, Penguin Books: 269-293.
Loveday, L. (1982). The Sociolinguistics of Learning and Using a Non-native Language. Oxford:
Pergamon Press.
Lyons, J. (1996). ‘On competence and performance and related notions’. In, G. Brown, K.
Malmkjaer and J. Williams (eds.): 11-32.
Makino, S. (1991). ‘ACTFL no gaikokugo nooryoku kijun oyobi soreni motozuku kaiwa nooryoku
tesuto no rinen to mondai’ (Issues in ACTFL proficiency guidelines and oral proficiency
interview). Sekai no Nihongo Kyooiku (Japanese-language Education around the Globe), 1:
15-32.
McNamara, T. (1995). ‘Modelling performance: opening Pandora’s box’. Applied Linguistics 16,
2: 159-179.
Nattinger, J. and J. DeCarrico (1992). Lexical Phrases and Language Teaching. Oxford: Oxford
University Press.
Oller, J. Jr. (1979). Language Tests at School: A Pragmatic Approach. London: Longman.
Oller, J. Jr. (1983), (ed.). Issues in Language Testing Research. Rowley, Mass: Newbury House.
Pawley, A. and F. Syder (1983). ‘Two puzzles for linguistic theory: nativelike selection and
nativelike fluency’. In, J. Richards and R. Schmidt (eds.): 191-226.
Purcell, E. (1983). ‘Models of pronunciation accuracy’. In, J. Oller Jr. (ed.): 133-153.
Raffaldini, T. (1988). ‘The use of situation tests as measures of communicative ability’. Studies in
Second Language Acquisition 10, 2: 197-216.
J. Richards and R. Schmidt (1983). Language and Communication. NY and London: Longman
Richards, J., J. Platt and H. Platt (1992). Dictionary of Language Teaching and Applied
Linguistics. Second edition. Harlow, Essex: Longman.
Savignon, S. (1983). Communicative Competence: Theory and Classroom Practice. Reading,
Mass: Addison-Wesley.
Schachter, J. (1990). ‘Communicative competence revisited’. In, B. Harley, J. Cummins, M.
Swain, and P. Allen (eds.): 39-49.
Shohamy, E. (1996). ‘Competence and performance in language testing’. In, G. Brown, K.
Malmkjaer and J. Williams (eds.): 138-151.
Upshur, J. and T. Homburg (1983). ‘Some relations among language tests at successive ability
levels’. In, J. Oller Jr. (ed.): 188-202.

1
As pointed out by Lyons (1996), competence and performance can be differentiated conceptually. This study thus
rests upon the assumption that the characteristics of communicative competence can be inferred from communicative
performance (see also Footnote Eighteen).
2
Canale and Swain (1980, 1981, Canale 1983) proposed that communicative competence consists of four major
components (general psycholinguistic factors such as memory limitations or perceptual strategies were excluded):
grammatical competence (knowledge of lexical items and rules of morphology, syntax, sentence-grammar semantics
and phonology), discourse competence (cohesion in form and coherence in meaning), sociolinguistic competence
(sociocultural rules of use) and strategic competence (communication strategies used to enhance communication and to
compensate for breakdowns in communication).
3
Bachman and Palmer’s sociolinguistic competence “...enables us to create or interpret language that is appropriate to a
particular language setting...” (Bachman and Palmer 1996: 70). It includes sensitivity to differences in dialects,
register, naturalness, and an ability to use cultural references and figures of speech.
4
Cziko (1984) differentiates between models of communicative competence which are descriptive (involving factors
which may or may not be psychologically independent, which appear to be involved in communicative competence)

11
 and those which are working models (involving uncorrelated components or combinations of components of
communicative competence).
5
Descriptive-output frameworks (such as ACTFL 1986, 1987 oral proficiency guidelines) are not considered suitable
as the basis for validating communication competence frameworks in this paper. They have limited applications to
second language pedagogy (that is, one needs to understand the inputs which make up fluency in order to teach fluency
to second language learners), and so they are not discussed further.
6
The terms `framework` and `model` are used deliberately to differentiate between conceptualisations of
communicative competence which have and have not been empirically validated.
7
The development of theories of communicative competence can progress significantly with descriptive validation
(Canale 1988), so both approaches to validation are considered in this paper.
8
Halliday (1973, 1978) proposed a model of language where competence in a language consisted of a system of
meaning (semantic) potentials, which relate behavioural potentials to lexico-grammatical potentials (hence the term
socio-semantic network).
9
“...a lexicalized sentence stem is a unit of clause length or longer whose grammatical form and lexical content is
wholly or largely fixed; its fixed elements form a standard label for a culturally recognized concept...most such units
are not true idioms but rather are regular form-meaning pairing...” (Pawley and Syder 1983: 191-192).
10
Halliday`s framework of communicative competence consisted of the overlapping of three components: textual
(linguistic factors including grammar, phonology, lexis and kinesics), ideational (semantic-notional relationships), and
interpersonal (pragmatic factors); and interpretive procedures where the three knowledge systems in communication
intersected (Halliday 1978, cited in Candlin 1986).
11
Whether this would in fact be an error or not would depend on the context and the individuals involved.
12
Orthogonal (or varimax) rotation assumes that the underlying factors are uncorrelated, whereas oblique rotation
assumes that the underlying factors are correlated.
13
Researcher’s own assessment based upon amount of formal study of Japanese by subjects and level of Japanese
displayed by subjects during interviews.
14
A number in line with other similar studies (for example, Bayard 1990, Gorden and Abell 1990).
15
The samples from the interviews were selected on the basis of being an extended piece of discourse by the subjects
and for their potential ‘interest’ value for the native speaker evaluators.
16
See Haugh (1998a: 140-142).
17
Although Bachman and Palmer do acknowledge the interactional nature of communicative competence with the
inclusion of strategic competence, they do not attempt to describe this interaction in any great detail.
18
The fact that communicative competence cannot be directly inferred from communicative performance (Chaulhoub-
Deville 1996, Shohamy 1996) makes the analysis of different test methods necessary.

12

View publication stats