Can Meta-Analysis Resolve the ESP Controversy? CHARLES AKERS: Metaanatysis is a quantitative, analytical approach (0 the review of scientific literature (Green and Hall 1984). The initial applications to psychology were in major areas of controversy, such as employment test validity (Schmidt and Hunter 1977), experimenter expectancy effects (Rosenthal and Rubin 1978), sex differences in conformity (Cooper 1979), and effects of psychotherapy (Smith and Glass 1977), Meta-analysis has become a popular technique for the exploration of many areas where experimental outcomes are in conflict, includ ing parapsychology (Honorton 1985; Hyman 1985). Meta-analytic methods do have some potential for resolving parapsycto- logical controversies. However, I will argue that there are serious limitations to what such methods can accomplish with old data—data on which skeptics and believers disagree vehemently. It is apparent that skeptics and believers do not agree on the quality of parapsychological studies (and hence on how much ‘weight each study should receive in the analysis). If this is true, then meta- analysis may be premature. It may be preferable to collect new data, under conditions prespecified by both parties to the debate, With research standards prespecified, the problem inherent to meta-analysis, of subjective bias in the coding of research quality, could be avoided. Before exploring the issue of quality coding, and possible sesolution of the issue, a brief account of meta-analytic methods is in order. The initial step in ‘meta-analysis (after delineation of the problem area) is a definition of the data base. This involves setting explicit eriteria for which studies should be included in the literature review and which should be excluded. The reviewer then defines the nature of the independent and dependent variables he wishes to consider. Methodological or design quality features can and should be included. All such study characteristics are then coded and assigned numerical values (if only for Prepared withthe support ofthe Hodgson Fund of Harvard Universiy. 1 thank George Hansen, Soh Palmer, Robert Rosenthal, and EH. Walker for thet eral comments on cari drafts onthe purpose of nominal classification). Once the quantification has been com- pleted, the data can be statistically analyzed, either by conventional techniques or by methods specific to meta-analysis. The usual goal is to determine how (or whether) study outcomes are correlated with or influenced by certain indepen- dent variables (though conclusions about causality may not be warranted). “Study outcome" can be defined in terms of some quantitative measure of effect size, such as Cohen's (197) d index. “The emphasis on quantification is one feature that distinguishes meta-analy- sis from a traditional, qualitative review. Another distinguishing feature of meta- analysis isthe attempt to make every step in the inferential process explicit. In @ qualitative review, the steps in the process are more likely to be implicit, re- Quiring the reader to make inferences. An advantage of meta-analysis is that, with all steps inthe inferential process made explicit, it should be easier than in 4 traditional review to identify sources of error or bias. If two meta-anaiysts disagree, it should be easier to identify the reasons for their disagreement. Because all variables must be quantified and all inferences made explicit, meta-analysis involves difficult or even arbitrary judgments. In a review of the various pitfalls inthe approach, M. J. Strube and D. P. Hartmana (1983) con- , ST:131-446,Charles Akers 625 ——. 1984. The Integrative Research Review: A Systematic Approach. Beverly Hills, Cali: Sage Davis, J, W.,and Akers, C. 1974, Randomization and tests for randomness, Journal of Parapsychology, 38:393-407. Dixon, N. F. 1981. Preconscious Processing. Chichester, England: Wiley. Feldman, S. E, 1980, Solomon E. Feldman comments on Edward W. Karnes et al, re remote viewing. Zeteic Scholar, No. 7:131 Fiske, D. W. 1983. The meta-analytic revolution in outcome research. Journal of Con- sulting and Clinical Psychology, $1:65-10, Girden, E. 1978. Parapsychology. In Handbook of Perception, vol. 10, edited by E. C Carterette and M. P. Friedman. New York: Academie Press Glass, G. V., B. McGaw, and M. L. Smith. 1981. Mera-analysis in Social Research, Beverly Hills, Calif: Sage Glucksberg, S. 1982. Not seeing is believing [review of Norman F. Dixon's Preconscious Processing). Contemporary Psychology, 27:856-858. Green, B. F.,and J. A. Hall. 1984, Quantitative methods for literature reviews. Annual Review of Psychology, 35:37-53. Hansel, C. E. M. 1980. ESP and Parapsychology. A Critical Re-Evaluation, Buttalo: Prometheus Books. Hoebens, P. H. 1980. Piet Hein Hoebens comments on Edward V. Karnes et al. re remote viewing. Zeteic Scholar, No. 7131-132 Honorton, C. 1977. Psi and internal attention states. In Handbook of Parapsychology, edited by B. B. Woman, 435-472, New York: Van Nostrand Reinhold. ——. 1979, Methodological issues in free-response psi experiments. Journal of the American SPR, 73:381-394. ——. 1985. Meta-analysis of psi ganzfeld research. Journal of Parapsychology. 4951-91 Hyman, R. 1981, Further comments on Schmidt's PK experiments: Alternative explana- tions are abundant. Skeptical Inquirer, 5(3}:34-40. 1985, The ganzfeld/psi experiment: A critical appraisal. Journal of Paraps)~ chology, 49:3-49, Johnson, M. 1975. Models of control and control of bias. European Journal of Para- psychology. 1, (1):36-44. Karnes, F. W. 1980. Edward W. Karnes replies to Solomon E. Feldman, Piet Hein Hoebens, and Evan Harris Walker. Zevetic Scholar, No. 7137-138. ——. 1981, Edward W. Karnes replies to Evan Harris Walker's above comments. Zetetic Scholar, No. 8128-130 Karnes, E. W., E. Susman, P. Klusman, and L. Turcotter. 1980. Failures to replicate remote viewing using psychie subjects. Zevetic Scholar, No. 6:66-16. Lord, C., L. Ross, and M. Lepper. 1979. Biased assimilation and attitude polarization: ‘The effect of prior theories on subsequently considered evidence. Journal of Per- sonality and Social Psychology, 37:2098-2108, Mahoney, M. J. 1977. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, :161-17S625 Can Meta-Analysis Resolve the ESP Controversy? Marks, D. and R. Kammann. 1980. Comments by David Marks and Richard Kammann {commentary on the paper by Professor Karnes etal]. Zeteric Scholar, No. 6:83-84, McClenon, J. 1982. A survey of elite scientists. Their attitudes toward ESP and paré- psychology. Journal of Parapsychology, 46:127-182 Merikle, P. M. 1982. Unconscious perception revisited, Perception and Psychophysics, 31:298201 Mintz, J. 1983, Inepsating research evidence: A commentary on meta-analysis, Jounal of Consulting and Clinical Pxychology. 5171-15 Morris, R. L. 1982. Review of Parapsychology: Science or Magic? A Psychological Perspective, by J. E. Alcock. Journal of the American SPR, 76:177-186. Moss, S., and D. C. Butler. 1978. The sciemtifie credibility of ESP. Perceptual and ‘Motor Skills, 46:1063-1078. Nicol, J. ¥. 1986. Some difficulties in the way of scientific recognition of extrasensory perception. in Exirasensory Perception, edited by G. E. W. Wolstenholme and E.C. P. Millar. New York: Citadel Press. Palmer. J. 1977. Altitudes and personality traits in experimental ESP research. In Handbook of Parapsychology, edited ¥y 8. B, Wolman, 175-201. New York: Van Nostrand Reinhold 1981. Methodological objections to the case for psi: Are formal controt con- divions necessary for the demonstration of psi? Paper presented at the convention of the American Psychological Association, Los Angeles, Calif, August 1983, Sensory contamination of free-response ESP targets: The greasy fingers hypothesis. Journal of the American SPR, 77:101-113. (in pres). Sensory identification af contaminaied free-rsponse ESP targets: Return of the greasy finger. Journal ofthe American SPR. Pratt, J. G., J. B. Rhine, B. Mott-Smith, C. E, Stuart, and J. A. Greenwood. 1966, Exura-Sensory Perception after Sixty Years. Boston: Bruce Humphries, (Originally publishes by Holt in 1940.) Price, G. R. 1955. Science and the supernatural. Science, 122:389-367, 1972, Apology to Rhine snd Soal, Letter to Science, 175:389. Puthoff, H. E.. and R. Targ. 1976, A pereeptual channel for information transfer over kilometer distances: Historical perspectives and recent research. Proceedings of the IEEE, 329-384, Rand Corporation, 1955. A Millon Random Digits. Glencoe, th: Free Press. Randi, J. 880a, Flim-Flam? The Trush about Unicorns, Parapsychologs. and other Delusions. New York: Lippincott & Crowell. [Revised edition published by Pro- metheus Books, Buffalo, N.Y, 1982.) 19800. Comments by James Randi. [commentary: on the paper by Professor Karnes etal. Zetetic Scholar, No. 684. Riess, BF. 1937. A case of high scores in card-guessing at a distance. Journal of Parapsychology, 1260-263. 1939, Further dats from a case of high scores in card-guessing, Journal of Parapsychology. 3:79-84 Rosenthal. R. 1986. Mesa-Analytc Procedures for Social Research. Beverly Hills, Cali Sage.Charles Akers 627 Rosenthal, R., and D. B. Rubin, 1978. Interpersonal expectancy effects: The first 345 ‘studies. Behavioral and Brain Sciences, 3:377-386 Sargent, C. L. 1980a. Comments on “Effects of associations and feedback on psi in the ‘anafeld.” Letter to the editor. Joural of the American SPR, 74:265-267, 19806. Exploring Psi in the Gansfeld. Parapsychological Monographs, No. 17, [New York: Parapsychology Foundation, Schmidt, F. L., and J. E, Hunter. 1977. Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62:329-S40. ‘Schmidt, H, 1980, A program for channeling psi data into the laboracory and onto the critic's desk. In Research in Parapsychology 1979, edited by W. G. Roll. Metuchen, Jz Searecrow Press ‘Smith, M. L.. and G. V. Glass. 1977. Meta-analysis of psychotherapy outcome studies. American Psychologist, 32:752-700. Soal, S. G., and F, Bateman, 1954. Modern Experiments in Telepathy. London: Faber and Faber ‘Strube, M.J., and D. P. Hartmann, 1983, Meta-analysis: Techniques, applications, and functions. Journal of Consulting and Clinical Psychology, $1:14-27 Tart, C.T. 1980. Comments by Charles T. Tart [commentary on the paper by Professor Kames et all. Zeretic Schalar, No. 685-6. ‘Terry, 4, and C. Honorton. 1976. Psi information retrieval in the ganzfeld: Two con- firmatory studies. Journal of the American SPR, 10:207-217 Walker, E, H. 1980, Evan Harris Walker comments on Edward W. Karnes etal. re remote viewing. Zetevie Scholar, No, 7:132-137, —. 1981. Evan Harris Walker replies. Zereule Scholar, 8:124-127, Wilson. R. 1966. Deviations from randomness in ESP experiments. International Journal of Parapsychology, 8387-295