OBSERVATIONAL STUDIES AND EXPERIMENTAL STUDIES IN THE INVESTIGATION OF CONFERENCE INTERPRETING Daniel Gile Universit Lumire Lyon 2 and

ISIT, Paris Published in Target 10:1.69-93. 1998 Abstract: In conference interpreting research (IR), empirical investigation can be classified as observational or experimental. The former can be used for exploration, analysis and hypothesis-testing, and is either interactive or non-interactive. Besides its conventional role of hypothesis-testing, the latter can be exploratory. The main methodological problems in both are related to validity and representativity and to quantification. At this stage, the most important contribution to interpreting research can be expected from observational procedures. Simple descriptive statistics and uncomplicated quantitative processing of the data still have much to offer. Rsum: La recherche empirique sur l'interprtation peut tre classe comme observationnelle ou exprimentale. La recherche observationnelle est interactive ou non et peut servir l'exploration, l'analyse et la vrification d'hypothses. Utilise gnralement pour la vrification d'hypothses, la recherche exprimentale peut galement tre exploratoire. Les principaux problmes mthodologiques dans ces deux dmarches se posent en matire de validit et de reprsentativit et propos de la quantification. A ce stade, c'est encore la recherche observationnelle qui prsente le plus important apport potentiel. Les statistiques descriptives et des mthodes quantitatives simples gardent un grand intrt. 1. Introduction As Translation Studies evolves into an academic (inter)discipline (Snell-Hornby, Pchhacker and Kaindl 1994, Wilss 1996), it has also been migrating from a predominantly 'philosophical' and prescriptive approach towards forms of investigation more consistent with the norms of research in more established disciplines (see Holmes, Lambert and Van den Broeck 1978, Toury 1995). At the same time, there is more empirical research as opposed to essay-type reflection, as exemplified inter alia by Tirkkonen-Condit's regular endeavors in this direction (Tirkkonen-Condit and Condit 1989, Tirkkonen-Condit 1991, Tirkkonen-Condit and Laffling 1993) and by the work on Think Aloud Protocols spearheaded by German investigators (see Krings 1986, Lrscher 1986, Knigs 1987, Kiraly 1995, Wilss 1996) - also see the collection of papers in Meta 41:1(1996) and Sguinot 1989. While the discipline is maturing, it is still vulnerable to various methodological weaknesses, as discussed in Toury 1991, 1995. Conference interpreting research (hereafter IR) is less mature and may be facing more fundamental questions about its identity and the way ahead (Gile 1995a, Gambier et al 1997). One fundamental decision to be made by investigators selecting empirical research projects is choosing between the experimental and observational paradigm. In IR, the present trend seems to favor experimental studies, thus possibly depriving the discipline of much useful data, and even of a large number of empirical research projects, since beginners and students who may be able and willing to carry out simple observational projects could hesitate to take on, or fail to complete experimental ones. This paper discusses methodological issues linked to this choice. 2. Definitions In the literature, a fundamental distinction is made between "theoretical research" which, roughly speaking, focuses on the intellectual processing of ideas, and "empirical research", which centers around the collection and processing of data. In principle, this distinction refers to different relative weights in a bipolar activity, not to the existence of two separate paradigms, one ignoring theory and the other ignoring data. D.Gile 1996 Obsvsexp page 1

Empirical investigation can be further classified into two broad categories: observational research, also called 'naturalistic', which consists in studying situations and phenomena as they occur 'naturally' in the field, and experimental research, in which situations and phenomena are generated for the specific purpose of studying them. The study of situations and phenomena initially generated for other purposes will be defined in this paper as observational. Classroom experiments will be considered experimental in this sense of the word if they are created for research purposes, and observational if they are initially designed for didactic purposes and are processed scientifically as an afterthought. As is the case in any classification, class boundaries can be fuzzy. For instance, having translators do their professional work in a room with a camera so that their activity can be observed with some precision (Sguinot 1989) could be considered an observational operation in spite of the controlled spatial environment, while having them work with a carefully controlled set of dictionaries and other documents could be enough to have the study cross the line into experimentation. However, boundary problems should not cause difficulty in the present discussion. As a footnote, one might add that in literary and historical studies, including those with a translation focus, the divisions explained above are less relevant, since research is based on pre-existing data and is therefore almost necessarily observational (though one could conceive of experimental setups in literary research). 3. Basic approaches in observational research Over the years, research methods have become increasingly sophisticated. While in sociology, in ethnology, in ethnolinguistics, in political science and in other disciplines, observational investigation is still the core approach, in other sciences, and in particular in psychology and in psychology-related disciplines, there has been a shift towards experimental procedures to the detriment of observational studies, which now tend to be viewed as preexperimental, and often as pre-scientific: "...exploratory activities that are not experimental are often denied the right to be classified as sciences at all" (Medawar 1981:69). In IR, after a long period of anecdote-, personal experience- and speculation-based writing, the research community is aspiring towards solid scientific status (see Arjona-Tseng 1989, Moser-Mercer 1991, Kondo and Mizuno 1995) and seems to be particularly sensitive to the scientific prestige factor. This may strengthen the attractiveness of the experimental paradigm, which has become prominent in conjunction with the recognition of the relevance of cognitive psychology to interpreting - cognitive psychology uses almost exclusively the experimental paradigm. As explained in the introduction, the predominance of the experimental approach may have deleterious effects. Most scientific disciplines need a descriptive foundation (cf. Beaugrand 1988:33, McBurney 1990:27). In interpreting research, there is not enough observational data yet, and Stenzl's call for more descriptive studies (1983:47) is still topical. It is therefore important to give such studies a chance by recalling their legitimacy as scientific endeavours per se even when they are not followed by hypothesis testing (cf. Fourasti 1966:134, Mialaret 1984:25, Grawitz 1986:430). Observational research can be broadly classified into three types of approach: 3.1 The exploratory approach In this paper, the adjective 'exploratory' refers to endeavors primarily concerned with the analysis of situations and events in the field without any prior intent to make a specific point, ask a specific question or test a specific hypothesis. Again, the popularity of experimental research in its hypothesis-testing variety (see below) has been detrimental to the exploratory approach, in that one often hears or reads that investigation around well-defined hypotheses is a sine qua non condition for "scientific work". This opinion is not shared by all (cf. Babbie 1992:91). Besides the authority argument, examples from mathematics, medicine and psychology show that good results can be obtained through the exploratory approach (also see Kourganoff 1971:54-55), though it must be stressed that: D.Gile 1996 Obsvsexp page 2

a. The absence of a well-defined and explicitly formulated hypothesis does not mean that exploration is done at random; some expectations always guide the investigator, b. Exploratory projects can, and in fact most often do lead to precise hypotheses and their subsequent testing. One example of such an exploratory approach can be found in a study of interpreting research productivity in the 80's and 90's undertaken by Franz Pchhacker (1995a,b). Pchhacker collected publication data, and then analysed it in terms of individual authors, of 'schools', of evolution over time, and drew a number of conclusions. He did not start with a specific hypothesis to test. 3.2 The focused analytical approach This title refers to research efforts focusing on a particular phenomenon through the analysis of observational data. This is the type of observational research most frequently encountered in IR. For example, Thiry (1975) sent out a questionnaire to the 48 interpreters classified as 'true bilingual' members of the International Association of Conference Interpreters in order to identify their biographical, professional and 'linguistic' idiosyncrasies. A more recent example is a study of how interpreting affected the role of participants in a war crime trial (Morris 1989). The idea was to analyse data obtained during the trial to see whether and how the interpreters' intervention actually changed the courtroom situation, especially as regards the role and interaction between the various actors. Another relatively recent analytical study is Gile's case study of his own terminological preparation for a medical conference and its 'efficiency' (1989, ch.14). 3.3 The hypothesis-testing approach This approach is similar to that usually found in experimental studies, in that the researcher seeks to collect data that will strengthen or weaken specific hypotheses about a particular phenomenon. The difference is that here the data are derived from field observation rather than experimentation, with the obvious limitations which arise from the fact that environmental conditions are given, not controllable, hence the risks associated with uncontrolled variability. Observational hypothesis-testing is rare in interpreting resarch. One example is Shlesinger's M.A. thesis (1989), in which she used field data to test the hypothesis that interpreting turns rather literate original speeches into more oral-like target speeches and viceversa. Observational studies can be methodologically simple. Generally speaking, as research in a field makes headway, observation and analysis tend to become finer and require more sophisticated tools and methods. In IR, the paucity of empirical research at this stage implies that the complexity of methods cannot be taken as an indicator for the relative value of their contribution: there are still many important aspects of interpretation to discover with simple methods. And yet, when such simple projects are suggested to beginners, they often balk at the idea and look for more ambitious projects which mostly turn out to be out of their reach. 3.4 Interactive and non-interactive observational research Another methodologically important distinction can be made between 'interactive' and 'non-interactive' observational research. The latter consists in observing situations and phenomena as they happen without the observed subjects playing an active role in the collection, analysis and assessment of the data (as opposed to interview-based and questionnaire-based studies), and with minimum data-distorting influence from the observer during the collection of the data. Actually, one could claim that there is always some interaction: the terms "interactive" and "non-interactive" are simplifications. It would be more appropriate, but less convenient, to talk of higher-interaction vs. lower-interaction. An example of such "non-interactive" or lower-interaction observational research is Strolz's doctoral dissertation (1992), in which she tested a number of hypotheses on the basis of the recordings of two interpreters' rendering of the same speech on the media. The two D.Gile 1996 Obsvsexp page 3

interpreters were not involved in the collection of the data; they only 'provided' the corpus by doing their job in a professional situation. Similarly, Williams's 1995 study of anomalous stress in interpreting consisted of an analysis of recordings of source texts and target texts in an 'authentic' conference. "Interactive" observational research in translation and interpreting studies generally involves participation of the investigator in the process under study and/or questionnaires or interviews, and thus entails the risk of interference from the researcher and/or a significant influence of the research procedure on the phenomenon under study, or the risk of interference from the subjects' personal perception, interpreting and reporting of facts. For example, Thiry's questionnaire was sent out to his 'true bilingual' colleagues who were also his competitors. Since the issue of language classification and in particular of 'true bilingualism' is a sensitive one in the conference interpreters' community, with concrete professional implications, it is difficult to rule out an influence of the very question on the responses on one hand, and on the investigator's interpretation thereof on the other. As to Gile's case study on conference preparation (1989), the knowledge that his preparatory work would be examined for research purposes is likely to have had some influence on the way he worked. The potential effects of such interaction may be taken on board usefully in the discussion of results if the direction of the bias is known or strongly suspected, as in the case of selfobservation of preparation for a conference (the assumption being that the preparation is done at least as seriously, and possibly more seriously, than under 'normal' circumstance). In other cases, its effects can be difficult to assess. Issues associated with interaction can be illustrated with a recent, rather large study on the expectations of users from conference interpreters (Moser 1995), based on user interviews performed by interpreters. Though the interviews followed a carefully designed questionnaire, one may question the validity and representativity of some of the findings for the following reasons (not to mention the fact that the interviewers were volunteers and had received written instructions, but no training - see Shipman 1988:85 on this subject): - It is a strong possibility that interpreters selected their respondents among users who seemed to show a friendly or otherwise positive attitude towards them or towards interpreting, rather than among indifferent or hostile delegates. - During the interview, the identity of the interviewers as interpreters was clear to the respondents, hence a possible effect on the answers given to the questions. - Because of the interpreters' subjective position on the matter, one cannot rule out bias in their interpretation of the responses. The risks associated with interaction in social anthropology, where the researcher's presence may influence the observed group's behaviour, have been acknowledged a long time ago. The numerous TAP (think-aloud-protocol studies) performed on translators over the past few years also entail a strong possibility of interaction between the research process and the translation processes under study (see for instance Toury 1991, Kiraly 1995 and Kussmaul 1995). 4. Experimental research: statistical hypothesis-testing vs. 'open experimenting'. Statistical hypothesis-testing is the most frequently used approach in experimental research in the social sciences. The procedure involves the selection of a sample of subjects who perform a task or are submitted to certain conditions in a 'controlled environment' in order to test a hypothesis on the basis of a 'statistical test' ('Chi square', 't test', 'F test', analysis of variance, etc.) showing 'significance' or 'non-significance' of measured and calculated values. The hypothesis is then either rejected or 'not rejected'. Most of the experimental research done on interpreting is of this type (see Gran and Taylor 1990, Lambert and Moser-Mercer 1994 and Tommola 1995). For many behavioral scientists, 'experimental research' has become a synonym for statistical hypothesis-testing. Researchers tend to forget that experiments can also be used for other purposes. In particular, they can have an exploratory function and help find answers to D.Gile 1996 Obsvsexp page 4

questions of the type "what will happen if..." (Fraisse, Piaget and Reuchlin 1989:I-92). Such experiments will be called 'open experiments' here. Again, present social norms make hypothesis-testing more appealing, and open experimenting appears as 'naive'. However, in industrial research, which is anything but naive, and in physics, chemistry and biology, which do not have to worry about their scientific status, researchers have no qualms about its use. The same applies to medicine, to agronomic research, and even to education research. It may also be worthwhile to recall that one of the main criticisms levelled against open experimenting, namely its inefficiency, attributable to the fact that the absence of a precise hypothesis to test leads to possible waste of time, material means and efforts, loses much of its relevance when computer simulation can be performed. Last but not least, advocates of the primacy of the hypothesis testing-paradigm point out repeatedly that "you only find what you are looking for", meaning that researchers need a precise hypothesis or research question to work on for fear of missing the significant phenomenon in a sea of data. The principle is generally true, although major discoveries have also been made at some distance from the researcher's focal point. On the other hand, focusing on one item may prevent the observer from noticing the presence of another, which may be important as well (see Reuchlin 1969:29-30). One example of open experimenting in translation research is a study in which an idea was presented graphically and subjects were asked to verbalise it in their native language (Gile 1995b, Ch. 3). The initial objective was to identify phenomena that could help justify some liberty in the production of the target text. Findings included a confirmation of interindividual and intra-individual variability in production, but also the identification of several types of information, partly language-dependent, that subjects added to the information they aimed to get across to their receivers (the 'message'), without necessarily being aware of it or controlling this added information. Open experimenting provides an observation platform which may then lead to more precise hypotheses and to statistical hypothesis-testing. In terms of experimental design, it is less difficult than the latter, because it requires less stringent controls (Fraisse, Piaget and Reuchlin 1989), but it does provide the basic advantages of experimental procedure as a whole, and in particular the possibility of having the same task performed under identical conditions by more than one person, something which in interpreting only rarely happens in the field. 5. Fundamental methodological issues in experimental and observational research in IR In principle, the major advantage of observational research is that it allows the investigation of phenomena as they occur naturally, with no distortion induced by the study. However, as already pointed out, this is not always the case, as many observational procedures do generate interference with the 'natural' phenomena (see also Shipman 1988). This is particularly true in interpreting: due to the evanescent nature of both input and output, on-line on site observation is often required, but when interpreters know they are being 'observed' for the purposes of close scrutiny, this may change their behavior, and not telling them is ethically problematic. The obvious way around the problem is to use data which are 'naturally' recorded and non-confidential, as is the case in TV interpreting, but samples thus obtained are not representative of all practitioners and all professional environments (see Kurz forthcoming). Another problem is associated with the diversity of working conditions (see section 6.2), which makes it difficult to identify the contribution of individual variables to the phenomena observed. Ideally, experimental research offers the solution to the problem: by creating an environment where all relevant factors are controlled, it is possible to 'isolate' and measure the effect of the variable under study (the 'experimental' or 'independent' variable). Unfortunately, the implementation of this principle is often far from simple. The following obstacles are particularly salient and have been generating much criticism among interpreters against experimental research:

D.Gile 1996

Obsvsexp

page 5

a. The experimental situation itself, being 'unnatural' (if only because the subjects know that they are being observed by researchers as opposed to being listened to by clients for remuneration within a professional framework), may generate processes somewhat different from the ones occurring in the natural environment (see section 6.1). Not enough data are available about differences in interpreting performance depending on the context being natural or experimental to decide that experiments do or do not distort processes significantly, but the possibility must be considered. b. The very selection of the relevant variables to be controlled is determined by subjective, non-scientific considerations, whenever not enough solid evidence is available (see for example Moravcsik 1980:46). As pointed out earlier, this is the case in IR. By controlling some less relevant variables and ignoring more important ones, investigators may well lose much of the potential benefit of the experimental paradigm. This is a serious risk in research performed by non-interpreters, who may be unaware of factors deemed relevant by interpreters and fail to control them. c. To address the issue of variability, experiments are done on samples. In interpreting, access to data and to subjects is problematic (see for example Gile 1995a), which leads to small, nonrandom samples, and to the possibility of interference from uncontrolled variables associated with the sampling procedure (see Section 6.2). 6. Methodological issues in the literature Though interpreting and translation investigators are vulnerable to a wide range of possible methodological weaknesses and errors, in the context of observational studies vs. experimental studies, three types of problems seem to be particularly salient in their research: 6.1 Weaknesses related to the validity of the data One of the most frequently encountered problems in IR has to do with the validity of the data as actual interpreting data. While in non-interactive observational research, data can be considered 'authentic', validity problems start with interactive observational research, precisely because responses elicited 'artificially' tend to be influenced by the way questions are put, by the interaction between the interviewer and the interviewee, and by the image of the interviewee or respondent that will be reflected back onto him by his own responses (see for instance Frankfort-Nachmias and Nachmias 1992). At an even more basic level, the subjective perception of 'reality' by respondents and interviewees can also be misleading. For instance, in Djean Le Fal 1978, it turned out that interpreters may perceive delivery speed as the number one difficulty in the interpretation of a speech read from a text, while the actual delivery rate is not high - other features of the speech make it appear to be high. Similarly, experiments by Collados Ais (1996) and Gile (1985 and 1995c) suggest that informants assessing the quality of an interpreter's linguistic output or its fidelity to the original can be very unreliable. This misleading nature of personal perception is one of the main reasons why scientific methods in general can be advocated in the field of interpreting despite the limitations of the scientific approach and the significant contribution of the 'personal theorizing' approach (Gile 1990). A validity problem that deserves special attention is associated with self-observation, as mentioned above. Not only is the analysis of mental processes through introspection obviously vulnerable to personal bias, but even attempts at more objective observational research on one's own work are probably less reliable than studies of other peoples' work. This does not imply that self-observation should be banned, as it offers advantages in terms of motivation, availability and access to a corpus. Nevertheless, attention to potential methodological problems arising in such research is called for. Validity problems become more severe in experimental research, especially in hypothesis testing and in research done by non-interpreters. Practitioners challenge the validity of data obtained with student interpreters and with amateurs with no training in interpreting as D.Gile 1996 Obsvsexp page 6

opposed to professionals (e.g. in Treisman 1965, Goldman-Eisler 1972, Lawson 1967, Chernov 1969, Barik 1971, Kopczynski 1980). They also express doubts as to the comparability of 'real' interpreting and the tasks given to subjects in experiments: interpreting written texts (Gerver 1974), some artificially composed with the aim of obtaining certain linguistic characteristics (Dillinger 1989), or even 'interpreting' random strings of words (Treisman 1965). Finally, as mentioned earlier, they have doubts as to the validity of data obtained in a laboratory situation where the speaker is not actually seen and where the interpreter does not 'feel the atmosphere' of a real conference. References to this issue are numerous: Politi 1989, Shlesinger 1989 (she rejected experimental procedures and later changed her mind and started performing experimental research herself), Dillinger 1990, Lambert 1994, Jrg 1995 (who reports that one of his subjects complained about the 'artificial' nature of the experiment). 6.2 Representativity of the data Another major methodological problem lies with the representativity of the data even when they are considered valid as interpreting data: - As already mentioned, samples in interpreting research are generally small, with a few exceptions in observational research such as Thiry's dissertation (1975), a Japanese study on 189 subjects (Yoshitomi and Arai 1991), and Moser's user expectation survey (1995). In small samples, the statistical procedure only partly smoothes out possible inter-individual variation. - The fact that samples are not random undermines the validity of statistical tests performed on them (Hansen et al. 1953, Snedecor and Cochran 1967, Charbonneau 1988, Babbie 1992:448), as non-random sampling does not exclude bias, and bias implies that sample means do not tend to approximate the population mean. The same problem is found in research in many other fields, in particular medicine and psychology, and the latter has traditionally come under criticism for studying not the human population at large but the population of first-year psychology students (Mc Nemar 1946, Smart 1966 and Sears 1986 are quoted on this point in Charbonneau 1988:333). An acceptable approximation can sometimes be found in the form of critically controlled convenience sampling, in which subjects are selected because they are easy to access but are screened on the basis of the researcher's knowledge of the field and (ideally) of empirical data derived from observation and experimentation. In interpreting, it is sometimes risky to make similar claims, as interpreters do have direct knowledge of the field, but are also highly involved, and may therefore lack objectivity when investigating issues on which they have strong views. - Interpreting conditions are diverse with respect to many parameters widely viewed as significant, such as the subject of the speech, previous knowledge of the subject by the interpreter, the syntactic and lexical make up of the speech, delivery parameters, interpreter fatigue, interpreter motivation, working conditions, etc. In particular, the source-language and target-language variables are poorly taken into account, as most experiments and observational projects are carried out with two languages only, and hardly ever with 'exotic' languages such as Chinese, Japanese, Arabic, Polynesian languages, etc. Each experiment only represents one, two or up to what can only be a minute fraction of the total number of possible combinations of values of the significant parameters. Generalising is therefore hazardous. - Another problem is linked to the possible variation over time in individual interpreting performance (the interpreter's "ups and downs"), a phenomenon intuitively perceived by interpreters, which may also limit the representativity of any single experiment. The amplitude of and precise reasons for such variations are not known, but the phenomenon is perceived as sufficiently salient by the practitioner's community to require caution. 6.3 Quantification Interpretation is a complex phenomenon which interpreters consider difficult to measure (see for instance Pchhacker 1994). Though they may accept the validity of a word or syllable count as a measure of 'linguistic throughput' (with reservations, as explained in Pchhacker D.Gile 1996 Obsvsexp page 7

1994), no such yardstick exists for the measurement of information or 'message' throughput. Neither have satisfactory ways been found to measure fidelity or the message reception rate among delegates, and fidelity indicators such as errors are challenged because of allegedly inadequate definitions (see for example Bros-Brann 1976 and Stenzl 1983). In this context, observational research is less vulnerable to errors than experimental research. This is obvious as regards the validity of data, but is also easily explained for the two other types of problems. Observational studies generally do not rely on quantification to the same extent as experimental studies, and tend to be content with rather gross order-ofmagnitude indications. Experimental studies of the hypothesis-testing type are based on mathematical comparisons, and uncertainty and variability in the measurement of the relevant values have more serious consequences. Furthermore, while such studies are designed for the purpose of drawing inferences on the population from the samples used (hence the importance of proper statistical procedure, including sampling methods), generalizations are not intrinsically part of observational research. When authors of such studies do generalize, their inferencing is intuitive rather than mathematical, and easier to assess for readers than in experimental research, because the data are known to be authentic rather than derived from an 'artificial' experimental environment. 7. Strategies 7.1 Validity of the data Validity problems in interactive observational interpreting research as described above are essentially the same as in questionnaire- and interview-based studies in other disciplines, and countering them is possible with the strategies and tactics developed for such studies in other fields such as sociology and psychology. Validity problems in experimental interpreting research are more difficult to deal with: too little solid data are available for assertions to be made regarding the comparability of sight translation data with interpreting data, of student performance with professional performance, of work into a regular active language with experimental interpreting into a usually passive language, etc. Under the circumstances, I believe that in research on interpreting (as opposed to linguistic, psycholinguistic or neurolinguistic research performed on interpreters for the purpose of investigating cognitive processes rather than interpreting per se), more weight should be given to the study of phenomena as they occur in the field, in other words to observational research. Experimental set-ups create 'artificial' environments by definition. Some would probably be considered an acceptable approximation for the phenomena under study (even Seleskovitch, the most outspoken anti-experimentalist, based her own dissertation on an experiment - cf. Seleskovitch 1975), while others have been rejected as non valid. The uncertainty zone between the two may be quite wide, and it would appear reasonable to try as much as possible to design experiments in which factors intuitively deemed relevant by practitioners are not ignored (see Dillinger 1990 for a challenge regarding the interpreters' doubts and an answer in Gile 1991). One constructive way of addressing the problem would be to compare experimental results and field performance, and study the amplitude and direction of possible differences: recordings of actual speeches can be used for experiments, and the product of their 'real life' interpretation and laboratory interpretation can be compared. As such comparisons accumulate, a solid foundation for assumptions on the comparability of experimental findings and interpretation in the field will gradually be constructed. Studies in which professional interpreters and students, amateurs and other subjects are given the same tasks (see for example Dillinger 1989 and Viezzi 1989) are also useful insofar as they provide data on comparability (or lack of comparability) in the performance of various categories of subjects and may thus open new avenues for extending virtual sample size. 7.2 Representativity of the data D.Gile 1996 Obsvsexp page 8

There is no easy solution to the problem of representativity, because of the small size of samples and the large number of different working environments and conditions, as well as the variety of individual interpreters' personal parameters. I believe the best way to address this problem is replication, in both experimental and observational procedures. Even if it is only partial (with somewhat different procedures and/or conditions), replication generates data comparable to at least some extent with the initial data, and thus may be said to 'increase' sample size, though admittedly not in the strict sense, which prevents the indiscriminate use of the data as if they came from a single sample (see Gay 1990:115). Unfortunately, at present, the number of interpreting investigators who perform experimental or observational research is very small (see Pchhacker 1995a,b), and all but a handful engage in a single effort for the purpose of obtaining a degree and then stop. Such investigators are interested in original research, much less in replication. The situation may change if interpreting schools establish research centers or research programs (see Gile 1995a). Another potentially useful resource for the assessment of data representativity in a study is the corpus of previously published studies, which may differ in their objectives and in conditions, but offer relevant data nevertheless. For instance, publications which contain transcripts of original speeches and their interpretation can be used to study the typology and distribution of errors, though in some, inferencing possibilities vary, depending on available information on how the data were obtained. 7.3 Quantification Admittedly, complex aspects of human behaviour such as interpreting are difficult to measure directly in a precise, thorough and monolithic way. There are, however, specific aspects of interpreting which are easier to quantify, such as the number and duration of pauses in the interpreter's delivery, the proportion of proper names correctly rendered, the number of lexical units in source and target texts and the relative frequency of specific lexical units in each, etc. Other aspects are more difficult to quantify. In a study on the quality of interpreting students' linguistic output (Gile 1987), the sensitiviity and criteria of native informants for deviations from acceptable linguistic usage was found to be highly variable (Gile 1985). Similarly, the quality of interpreting is difficult to assess accurately, as stressed inter alia by Pchhacker (1994) and demonstrated in Collados Ais (1996). Measurements can be inaccurate because of a lack of precise definitions (in error counts, quality assessment, etc.), because of observational inaccuracies or errors (due to personal bias or shifts in attention - see Gile 1995c), and because of the natural variability of the phenomena under study. In cases where such inaccuracies are likely, this must be taken into account when drawing conclusions. For instance, in a study on interpreting, error rates for two groups were respectively 27.22 and 23.45 (see Politi 1989, reporting on a study by Balzani), and the difference, about 15%, was checked for statistical significance. One may wonder about the usefulness of such statistical significance, considering the variability and possible non-random inaccuracies that may have crept into this small-sample experiment, including possible variability in error identification (in her 1983 M.A. thesis, Stenzl reports that when she applied Barik's error criteria to the corpus and tried to replicate the procedure, she found different results from his - see Barik 1971). The effect of a single outlier in a small sample can be large, as pointed out by Jrg (1995) in his data from a sample of 6 interpreters. No matter how powerful and refined new statistical methods are, they cannot overcome the fundamental limitations associated with high variability in small non-random samples. On the other hand, such inaccuracies and the limitations they entail need not be a major problem in research projects. Less accurate quantification can be performed in many cases using the tools that have been developed for similar problems in the social sciences, and be sufficient to provide the required answers. For instance, in a study on the deterioration of linguistic output quality during interpreting exercises (Gile 1987), in spite of poor informant reliability, the existence of 3 distinct levels of linguistic deviation frequency depending on the type of exercise (simultaneous interpreting, consecutive interpreting and ad-libbed presentations) could be established and proved robust when submitted to sensitivity analysis D.Gile 1996 Obsvsexp page 9

(sensitivity analysis checks to what extent results and conclusions change when relevant parameters vary within specified intervals, as can happen inter alia because of variability in human evaluation). Similarly, when trying to decide whether proper names pose a 'real' problem in interpreting, the point will have been made regardless of whether 80, 90 or 100% of the names in a speech are incorrectly rendered in the experiment (Gile 1984). If the question is whether or not interpreters lose some information in simultaneous under given conditions, the result is significant regardless of whether 10, 20 or 30 errors are detected in a half-hour interpreting segment (Gile 1989 - also see Babbie 1992:127). Here as elsewhere, common sense is paramount, and more important than statistical techniques (see Medawar 1981:27,93, McBurney 1990:5). One might also recall in this respect that 'statistically significant' differences may have no practical significance: the 20% difference between an average of 10 and an average of 12 errors per half an hour of interpreting may not be enough to provide 'significantly better' interpreting to a client; the 30% difference between having an average of 150 and 200 new terms to learn while preparing for a technical conference may not change significantly the difficulty of the endeavor or the choice of appropriate interpreting strategies. Most of the debates in interpreting research still focus not on minute differences, but on basic principles of 'doctrine': does work into one's non-native language produce a target speech of significantly poorer quality than that produced in one's native language? Should consecutive interpreting be taught for a whole year before training in simultaneous interpreting starts? Are shadowing exercises useful ?Small random differences between conditions do not contribute much to the debate, while wide, regular differences can have practical significance. Interpretation being largely unexplored, I believe that such wide, regular differences can still be found with simple quantitative methods, and advocate their use rather than more complex techniques which may yield less reliable data. Science proceeds by "successive approximations" (Kourganoff 1971, Camus 1996:8). First approximations, achieved with simple methods, are necessary to provide the basis for finer approximations, which will eventually require more sophisticated methods. At this stage, due to the paucity of data on variability and on co-variability between a larger number of potentially important variables, descriptive statistics (which describe phenomena) may be a more appropriate tool than inferential statistics (which attempt to make mathematics-based inferences on the population from the data collected on samples). Explicit descriptive statistics enable readers of published studies to decide for themselves whether samples and conditions are representative and whether results are significant, while statistical tests may give an undeserved seal of 'scientific objectivity' to a procedure which is not valid by strict standards (see Gore and Altman 1982). 8. Observational studies vs. experimental studies in translation research Similar issues exist in translation research, but their relative weights are different. The following are some differences which can have implications on research strategies, and in some cases on the choice of observational vs. experimental studies: 8.1 Factors facilitating observational research: - The product of translation, basically a text, is easy to store and readily available to investigators on a large scale at little cost, especially as regards literary translation. The product of interpreting, composed of spoken discourse plus body language (a "hypertext" Pchhacker 1994), is evanescent. Its observation requires recordings, preferably of both image and sound. There are relatively few recordings of interpreted speeches, and existing ones are difficult to access. - Technically speaking, translated texts are easier to process, with the possibility of rapid visual access to all parts of the corpus, while recordings of interpreted speeches require timeconsuming linear listening, or equally time-consuming transcription work, which inevitably leads to a loss of information. From this viewpoint, the experimental approach, which involves a 'controlled' limited corpus, is likely to be preferred by interpreters. D.Gile 1996 Obsvsexp page 10

- In translation, the time elapsed between the moment the translator reads a source-text segment and the moment s/he produces its final target-language version can last anywhere from a few seconds to hours, days or weeks. A corollary is that in field conditions, reactions to specific stimuli (such as manipulated independent variables) can be offset by further thinking and strategies or 'drown' in the general noise associated with the operation, and be undetectable to the researcher. This makes experimental procedures more difficult to conduct when studying the translation process. The TAP paradigm is a clever attempt to address the problem, but it is not without its weaknesses (cf. Toury 1991). In simultaneous interpreting, the EVS (ear-voice span, or time elapsed between the moment the interpreter detects a speech segment and the moment s/he produces its target-language reformulation) rarely exceeds a few seconds, and reactions to stimuli are likely to be more salient. 8.2 Factors facilitating experimental research: - Translators are numerous and are used to their product being exposed, as opposed to interpreters, whose population is much smaller, who feel vulnerable and who dislike exposure. Access to samples is therefore easier in translation. - 'Authentic' replication of an interpreting task in the field is essentially limited to speeches made during international events which are interpreted by several interpreters and broadcast over several radio and television channels. The validity of any further replication in the laboratory can be challenged because of the 'artificial' environment. In translation, it is possible to generate 'authentic' replication of translated tasks, for instance by commissioning the translation of the same text into the same language from several translators or translation agencies. The cost of a similar operation replicating field conditions in interpreting would be much higher. - In translation, text manipulations are easy to perform without the subject noticing them and without jeopardizing the validity of the translation task. In interpreting, text manipulations have been criticized for possibly generating interpreting behavior inconsistent with 'authentic' behavior. - Translation performance is less dependent on cognitive load management than interpreting performance, in which phenomena lasting a fraction of a second can lead to major irreversible consequences (Gile 1995a,b). It follows that: (1) Strict control of environmental conditions is less critical in translation than in interpreting. A gradual increase of sample size over time in several locations and under slightly different conditions is therefore less problematic. (2) Intra-individual performance variability is probably lower in translation then in interpreting, and may well lead to solid conclusions on the basis of smaller samples. The preceding analysis underscores two major differences between translation and interpretation as regards research possibilities: - On the whole, access to data on the product and their analysis are easier in translation. - The observation of the process is more difficult in translation. This may offer a partial explanation for the fact that translation is far ahead of interpretation in observational product-centered studies, while it is struggling with processoriented studies. 9. Conclusion Observational and experimental studies should be seen as mutually reinforcing, not mutually exclusive. Somes issues are better addressed by the observational paradigm, and some by experimental investigations, especially in fields where a considerable corpus of research already exists, as is the case of neurophysiology. This paper does not challenge the basic advantages of the experimental paradigm, nor does it advocate the choice of this or that method in any individual research endeavor. It does claim that in IR, there is a lack of descriptive data obtainable through simple methods, and that this weakens the power of more complex methods. In problem-solving oriented research disciplines such as mathematics, medicine, chemistry, etc., self-regulation occurs naturally as a function of results. In the D.Gile 1996 Obsvsexp page 11

behavioral sciences, including IR, it is more difficult to assess the value or 'truthfulness' of research findings, and paradigmatic directions are to a larger extent a function of social norms and research policy. Underlining problematic aspects of sophisticated methods and stressing the value of simple methods may encourage leaders of the IR community to shift somewhat the balance of efforts in favor of the latter, in particular among students working on a graduation thesis, which may help fill the existing gap in descriptive data. References Arjona-Tseng, Etilvia.1989. "Preparing for the XXIst Century". Proceedings of the Twentieth Anniversary Symposium of the Monterey Institute of International Studies on The Training of Teachers of Translation and Interpretation. Monterey. pages not numbered. Babbie, Earl. 1992. The practice of social research. Belmont, California: Wadsworth. Sixth Edition. Barik, Henri. 1971. "A description of various types of omissions, additions and errors encountered in simultaneous interpretation". Meta 15:1. 199-210. Beaugrand, Jacques P. 1988. "Dmarche scientifique et cycle de la recherche". Robert, ed. 1988: 1-35. Bros-Brann, Eliane. 1976. "Critical Comments on H.C. Barik's "Interpreters Talk a Lot, Among Other Things"". Bulletin de l'AIIC 4:1. 16-18. Camus, Jean-Franois. 1996. La psychologie cognitive de l'attention. Paris: Armand Colin. Charbonneau, Claude. 1988. "Problmatique et hypothses d'une recherche". Robert, ed. 1988: 66-77. Chernov, Ghelly V. 1969. "Linguistic Problems in the Compression of Speech in Simultaneous Interpretation". Tetradi Perevodchika Vol.6: 52-65. Collados As, ngela. 1996. La entonacin montona como parametro de calidad en interpretacin simultnea: la evaluacin de los receptores. Unpublished doctoral dissertation. Universidad de Granada. Djean Le Fal, Karla. 1978. Lectures et improvisations: incidences de la forme de l'nonciation sur la traduction simultane. Unpublished doctoral dissertation. University of Paris III. Dillinger, Michael L. 1989. Component Processes of Simultaneous Interpreting. Unpublished PhD dissertation. McGill University, Montreal. Dillinger, Michael L. 1990. "Comprehension during interpreting: What do interpreters know that bilinguals don't ?" The Interpreter's Newsletter 3: 41-55. Fourasti, Jean. 1966. Les conditions de l'esprit scientifique. Paris: Gallimard. Fraisse, Paul, Jean Piaget and Maurice Reuchlin. 1989. Trait de psychologie exprimentale. Vol. I. Histoire et mthode. Paris: PUF. Sixth Edition. Frankfort-Nachmias, Chava and David Nachmias. 1992. Research Methods in the Social D.Gile 1996 Obsvsexp page 12

Sciences. London, Melbourne, Auckland: Edward Arnold. Fourth Edition. Gambier, Yves, Daniel Gile and Christopher Taylor, eds. 1997. Conference Interpreting: Current Trends in Research. Amsterdam/Philadelphia: John Benjamins. Gay, L.R. 1990. Educational Research. Competencies for Analysis and Application. New York: Merril, MacMillan Publishing Company. Third Edition. Gerver, David. 1974. "The effects of noise on the performance of simultaneous interpreters: accuracy of performance". Acta Psychologica 38: 159-167. Gile, Daniel. 1984. "Les noms propres en interprtation simultane". Multilingua 3:2. 79-85. Gile, Daniel. 1985. "La sensibilit aux carts de langue et la slection d'informateurs dans l'analyse d'erreurs: une exprience". The Incorporated Linguist 24:1. 29-32. Gile, Daniel. 1987. "Les exercices d'interprtation et la dgradation du franais: une tude de cas". Meta 31:4. 363-369. Gile, Daniel. 1989. La communication linguistique en runion multilingue, Les difficults de la transmission informationnelle en interprtation simultane. Unpublished PhD dissertation. Universit Paris III. Gile, Daniel. 1990. "Scientific Research vs. Personal Theories in the investigation of interpretation". Gran and Taylor, eds. 1990: 28-41. Gile, Daniel. 1991. "Methodological Aspects of Translation (and Interpretation) Research". Target 3:2. 153-174. Gile, Daniel. 1995a. Regards sur la recherche en interprtation de confrence. Lille: Presses Universitaires de Lille. Gile, Daniel. 1995b. Basic Concepts and Models for Interpreter and Translator Training. Amsterdam/Philadelphia: John Benjamins Publishing Company. Gile, Daniel. 1995c. "Fidelity Assessment in Consecutive Interpretation: An Experiment". Target 7:1. 151-164. Goldman-Eisler, Frieda. 1972. "Segmentation of input in simultaneous interpretation", Psycholinguistic Research 1. 127-140. Gore, Sheila M. and Douglas G. Altman. 1982. Statistics in Practice. London: British Medical Journal. Gran, Laura and Christopher Taylor, eds. 1990. Aspects of applied and experimental research on conference interpretation. Udine: Campanotto Editore. Grawitz, Madeleine. 1986. Mthodes des sciences sociales, Paris: Dalloz. Hansen, M., W. Horwitz and W. Madow. 1953. Sample Surveys: Methods and Theory. New York, London, Sydney: John Wiley and Sons. Holmes, James S., Jos Lambert and Raymond van den Broeck, eds. 1978. Literature and Translation: New Perspectives in Literary Studies. Leuven: acco. D.Gile 1996 Obsvsexp page 13

Jrg, Udo. 1995. Verb anticipation in German-English simultaneous interpreting. Unpublished M.A. dissertation. University of Bradford. Kiraly, Donald. 1995. Pathways to Translation. Pedagogy and Process. Kent, Ohio and London, England: The Kent State University Press. Kondo, Masaomi and Akira Mizuno. 1995. "Interpretation Research in Japan". Target 7:1. 91106. Knigs, Frank G. 1987. "Was beim bersetzen passiert". Die Neueren Sprachen 86. 162-185. Kopczynski, Andrzej. 1980. Conference Interpreting (Some Linguistic and Communicative Problems). Nauk, Poznan: Wydawn. Kourganoff, Vladimir. 1971. La recherche scientifique, Paris:PUF. Krings, Hans P. 1986. Was in den Kpfen von bersetzern vorgeht. Tbingen: Gunter Narr. Kurz, Ingrid. forthcoming. "Getting the Message Across: Simultaneous Interpreting for the Media". Proceedings of the EST conference in Prague, September 1995. Kussmaul, Paul. 1995. Training the Translator. Amsterdam/Philadelphia: John Benjamins Publishing Company. Lambert, Sylvie. 1994. "Foreword". Lambert, Sylvie and Barbara Moser-Mercer, eds. 1994: 5-14. Lambert, Sylvie and Barbara Moser-Mercer, eds. 1994. Bridging the Gap: Empirical research on simultaneous interpretation. Amsterdam and Philadelphia: John Benjamins Publishing Company. Lawson, Everdina. 1967. "Attention and Simultaneous Translation". Language and Speech 10:1. 29-35. Lrscher, Wolfgang. 1986. "Linguistic aspects of translation processes". House, J. and S. Blum-Kulka, eds. 1986. Interlingual and intercultural communication. Tbingen: Gunter Narr: 277-292. McBurney, Donald H. 1990. Experimental Psychology. Belmont, California: Wadsworth. Mc Nemar, Q. 1946. "Opinion-attitude methodology". Psychological Bulletin 43. 289-374. Medawar, Peter B. 1981. Advice to a Young Scientist. London and Sydney: Pan Books. Meta 41:1. 1996. Special issue on Translation processes, guest-edited by Frank Knigs. Mialaret, Gaston. 1984. La pdagogie exprimentale. Paris: PUF. Moravcsik, Michael J. 1980. How to Grow Science. New York: Universe Books. Morris, Ruth. 1989. The impact of interpretation on legal proceedings. Unpublished M.A. thesis. Communications Institute, Hebrew University of Jerusalem.

D.Gile 1996

Obsvsexp

page 14

Moser, Peter. 1995. Survey on Expectations of Users of Conference Interpretation. Final Report. produced by SRZ Stadt + Regionalforschung GmbH, Vienna. Mimeographed. English translation. Moser-Mercer, Barbara. 1991. "Paradigms gained or the art of productive disagreement". AIIC Bulletin 19:2. 11-15. Pchhacker, Franz. 1994. Simultandolmetschen als komplexes Handeln. Tbingen: Gunter Narr. Pchhacker, Franz. 1995a. ""Those Who Do...": A Profile of Research(ers) in Interpreting". Target 7:1. 47-64. Pchhacker, Franz. 1995b. "Writings and Research on Interpreting: A Bibliographic Analysis". The Interpreter's Newsletter 6. 17-32. Politi, Monique. 1989. "Le signal non verbal en interprtation simultane". The Interpreter's Newsletter 2. 6-10. Reuchlin, Maurice. 1969. Les mthodes en psychologie. Paris: PUF. Robert, Michle, ed. 1988. Fondements et tapes de la recherche scientifique en psychologie. St-Hyacinte, Qubec: Edisem and Paris: Maloine. Schjoldager, Anne. 1995. "Interpreting Research and the 'Manipulation School' of Translation Studies". Target 7:1. 29-45. Sears, D.O. 1986. "College sophomores in the laboratory: infuences of a narrow data base on social psychology's view of human nature". Journal of personality and social psychology 51. 515-530. Sguinot, Candace. 1989. "The Translation Process: An Experimental Study". Sguinot, Candace, ed. 1989. The Translation Process. Toronto: H.G. Publications, School of Translation, York University. Seleskovitch, Danica. 1975. Langages, langues et mmoire. Paris: Minard. Shipman, Martin. 1988. The Limitations of Social Research. London and New York: Longman. Third Edition. Shlesinger, Miriam. 1989. Simultaneous Interpretation as a Factor in Effecting Shifts in the Position of Texts on the Oral-Literate Continuum. Unpublished M.A. thesis. Tel Aviv University. Smart, R. 1966. "Subject selection bias in psychological research". Canadian psychologist 7. 115-121. Snedecor, George W. and William Cochran. 1967. Statistical Methods. Ames, Iowa: The Iowa State University. Sixth Edition. Snell-Hornby, Mary, Franz Pchhacker and Klaus Kaindl, eds. 1994. Translation Studies An interdiscipline. Amsterdam/Philadelphia: John Benjamins Publishing Company. Stenzl, Catherine. 1983. Simultaneous Interpretation - Groundwork towards a D.Gile 1996 Obsvsexp page 15

Comprehensive Model. Unpublished M.A. thesis. University of London. Target 7:1. Special Issue on Interpreting Research. John Benjamins Library. Strolz, Birgit. 1992. Theorie und Praxis des Simultandolmetschens. Unpublished PhD Dissertation. Geisteswissenschaftliche Fakultt der Universitt Wien. Thiry, Christopher. 1975. Le bilinguisme chez les interprtes de confrence professionnels. Unpublished PhD dissertation. University of Paris III. Tirkkonen-Condit, Sonja, ed. 1991. Empirical Research in Translation and Intercultural Studies. Tbingen: Gunter Narr Verlag. Tirkkonen-Condit, Sonja and John Laffling, eds. 1993. Recent Trends in Empirical Translation Research. University of Joensuu. Faculty of Arts. Tirkkonen-Condit, Sonja and Stephen Condit, eds. 1989. Empirical Studies in Translation and Linguistics. University of Joensuu. Faculty of Arts. Tommola, Jorma, ed. 1995. Topics in Interpreting Research. University of Turku. Centre for Translation and Interpreting. Toury, Gideon. 1991. "Experimentation in Translation Studies: Achievements, Prospects and some Pitfalls". Tirkkonen-Condit, Sonja, ed. 1991: 45-66. Toury, Gideon. 1995. Descriptive Translation Studies and Beyond. Amsterdam and Philadelphia: John Benjamins Publishing Company. Treisman, Anne. 1965. "The Effects of Redundancy and Familiarity on Translation and Repeating Back a Native and Foreign Language". British Journal of Psychology 56. 369-379. Viezzi, Maurizio. 1989. "Information retention as a parameter for the comparison of sight translation and simultaneous interpretation: an experimental study". The Interpreter's Newsletter 2. 65-69. Williams, Sarah. 1995. "Observations on Anomalous Stress in Interpreting". The Translator 1:1. 47-64. Wilss, Wolfram. 1996. Knowledge and Skills in Translator Behavior. Amsterdam/Philadelphia: John Benjamins Publishing Company. Yoshitomi, Asako and Kiwa Arai. 1991. "Tsyaku katei ni kansuru kis jikken" ("A basic experiment on the interpreting process"), in Watanabe, Shoichi, ed. 1991. Mombush josei kagaku kenky: gaikokugokyiku no ikkan to shite no tsyakuysei no tame no kyikunaiyhh no kaihatsu ni kansuru sgteki kenky (Report on a Project funded by the Ministry of Education: A Comprehensive Study of the Development of Interpretation Training Methodology as a Part of Foreign Language Training). Unpublished manuscript. Sophia University, Tokyo. 373-439.

D.Gile 1996

Obsvsexp

page 16