Achen

27

Two Cheers for Charles Ragin

Christopher H. Achen
Ragins Three Claims For almost two decades, Charles Ragin has argued for a new methodological style in the social sciences. The detail and sophistication of his case have advanced during that period, but his fundamental claims have not changed. He argues that: Claim 1: Case-study analysis has a methodology of its own, and there is much to learn from it that can be learned in no other practical way; Claim 2: Quantitative research as most researchers practice it does not match the character of serious case-study methods and, more importantly, does not match social reality; and Claim 3: In light of Claims 1 and 2, conventional quantitative methods should be de-emphasized in favor of a new methodology that Ragin has developed, Qualitative Comparative Analysis (QCA), which aims to replicate the logic of casestudy analysis in a mathematical framework different from statistical theory. No knowledgeable person disputes Claim 1. Thus I begin with Claim 2, and then turn to Claim 3, where Seawright (2005) has directed his examination. Ragins Second Claim Ragin argues that social events often involve multiple conjunctural causation (1987: 20), and that linear statistical models fail to capture this aspect of social complexity:
Christopher H. Achen is professor of politics at Princeton University. His research focuses on political methodology, particularly as it applies to empirical democratic theory, American politics, and international relations. He is the author of Interpreting and Using Regression (1999) and The Statistical Analysis of Quasi-Experiments (1987), and co-author of Cross-Level Inference (1994). Studies in Comparative International Development, Spring 2005, Vol. 40, No. 1, pp. 27-32.

28

Studies in Comparative International Development / Spring 2005

In some contexts a certain cause may be relevant to a given outcome; in others it may be unimportant; and in still others the absence of this condition may be causally significant to the outcome . . . . Those who use statistical methods often must assume, for example, that the effect of a cause is the same across different contexts. (1987: 166, 167)1

Ragin implicitly concedes that some statistical methods can handle multiple conjunctural causation, but he argues that quantitative researchers regularly fail to use them, employing canned statistical procedures instead:
Investigators are not limited to simple, linear additive models; in practice, however, they usually stick fairly close to such formulations. If any tests for interaction or for population differences are performed, they are very simple in nature. (1987: 67)

There can be no doubt that Ragin is largely correct here. Many quantitative applications in the social sciences are dreadfully dumbed down, imposing linearity and other substantive assumptions that everyone knows are false, including the scholars doing the analyses. I have complained about atheoretical and substantively inappropriate linearity assumptions for more than a decade (Achen, 1992, 2002, 2005). Many others have long made similar and related points (for example, Berk, 2004 and the references cited therein). But the problem continues. Econometrics texts stress the linear case, and statistical packages make it easy for lightly trained scholars to carry out the calculations. Even among the more sophisticated users, simpleminded linearity assumptions are too often combined with prestigious statistical computing (maximum likelihood! Bayesian applications of Markov-chain Monte Carlo!) to stupefy the innocent. Of course, powerful statistical methods are a good thing. I like them myself; theyre fun. The point, however, is that substantively ill-informed use of them is not a good thing. In particular, elaborate likelihood functions with unthinking linear components (for example, unjustified linear link functions) are flim-flam. When one encounters them in published work, one ought simply to stop reading and turn to the next article. People will say that education [or social capital, or party competition, or . . . ] usually matters in these regressions, thinking that they have learned something. But as Ragin notes repeatedly in his writings, virtually all such variables matter sometimes, not other times, and only in conjunction with certain factors at still other times. The task is to learn which case falls into which causal pattern. But after reading all the regressions, we rarely know. There is just not a great deal to learn from the kind of simplistic linear statistical work that too often turns up in the journals. Thus, to me, Ragins Claim 2 has long been persuasive. As an early advocate of it, he deserves all the approbation and thanks that have come his way. Seawright (2005), too, joins in the applause for Ragins contribution. Ragins Third Claim Where Ragin and Seawright part company is at Claim 3. Ragin argues that statistical methods cannot cope with much of social reality. His view (2005; see also 1987:

Achen

29

6263) is that regression analysis is designed to attach to each variable an independent, context-free, net effect. He also argues that statistical populations rarely change during quantitative research (1987: 16, 55; 2000: 3738, 4550). And third, he writes:
While it is clear that users of regression analysis can utilize case-oriented knowledge to aid their attempts at causal inference, it is relatively rare for those who seek to make causal inferences on the basis of formal tests to utilize such knowledge. (2005: 31)

Unfortunately, these and many others of Ragins assertions about statistical analysis will sound peculiar to a quantitative researcher. For the truth is that only researchers with shallow training do quantitative, variable-oriented research the way Ragin describes it. Even harsh quantitative critics of regression applications would not characterize the field as Ragin does. What his descriptions actually show is just how primitive and gauche is the level of variable-oriented research he has in mind. The real McCoy is quite different, and in fact, resembles quite closely what Ragin himself recommends. For example, he (1987: chap. 3) argues that his method encourages researches to look at all cases individually when the sample is small, to investigate deviant instances and attempt to account for them, to interpret causal effects conditional on the observed cases and not on some imaginary sample, to look for contextual and holistic effects, and to go back and forth between model and data in a rich dialogue (1987: 52). But of course, this is precisely what the many good books on statistical data analysis (not statistics or econometrics, but data analysis) show the reader how to do.2 And it is that methodologyusing graphics, cross-tabulation, interaction effects, nonlinear fits, alternative sample definitions and populations, and all the restthat has created the rich legacy deriving from the great names in empirical social science, beginning with the earliest pioneers and extending to Stouffer, Lazarsfeld, Key, Converse, and others down to the present, as Ragin himself notes (2000: 52, fn. 7). Thus, Ragins idea that regression and other statistical methods are restricted to context-free net effects is a mystifying claim, indeed. Of course, as I said above, much empirical research fails to meet its own standards. But that is just because it is accepted widely and taught everywhere. If QCA becomes equally popular, one can be sure that many of its applications will be perfunctory, slapdash, or otherwise unimpressive, too. As Seawright (2005) remarks, the issue is not average work, but best practices. And on that criterion, Ragins description of regression analysis is simply not true to best practices, as he repeatedly concedes. In its current incarnation, Ragins (2000) methodology has become able to handle ordinal data and other aspects of contemporary social science data. But this is where the importance of Seawrights (2005) contribution lies. Each of the steps Ragin uses in his QCA methodology corresponds to their long-established quantitative equivalents. As Seawright shows, there is nothing genuinely new in Ragins quantitative techniquesthe central features reduce to regression analysis. Other clever and valuable aspects of Ragins method that do not reduce to regression have also

30

Studies in Comparative International Development / Spring 2005

long been done in statistical methodology, as the trained quantitative researcher will recognize. See, for example, Rosenbergs (1968) classic text on the interpretation of discrete data, Bishop et al.s (1977) sophisticated statistical justification of the rich dialogue with data, and the search for nonlinear and interactive effects that Rosenberg and other capable quantitative scholars exemplify. Recent developments continue to expand the power of statistical methods to cope with complex causal situations of precisely the sort Ragin has in mind (see especially Braumoeller, 2003). Because statistical techniques have drawn contributions from so many scholars, they are often far more powerful than Ragins QCA. For example, case-study methods are often much better at generating interesting hypotheses than statistical techniques, but much less good at assessing the reliability of their findings, as many scholars, including Ragin (2000: 29), have remarked. Standard errors are helpful, but only quantitative methods produce them. All that potent assistance to sound inference is lost when boutique mathematical methodologies replace recognized statistical theory. Thus, the actual implication of Ragins persuasive Claims 1 and 2 is not, as he believes, that statistical data analysis should be de-emphasized in favor of QCA when social causation is complex and the units of analysis are few.3 The real implication is that when combined with case studies or any other kind of evidence, clumsy and unthinking statistical data analysis should be abandoned. Nothing follows about the need for QCA. Hence Seawright is correct to argue that Ragins Claim 3 goes much too far. Conclusion For all my concerns about the direction in which Ragin wants to go, I join Seawright in stressing that Ragins achievements are considerable, a point I have made in print elsewhere (Achen, 1992). His was an early voice crying in the wilderness about the sterility and uselessness of much mindless linear regression, and his argument applies just as well to linear logit and probit, and to other linear maximum likelihood and Bayesian models. Very little in the social world fits these thoughtless specifications. When they are combined with mechanical significance testing, as they so often are, the result is usually a dreary, pointless exercise, no matter how long the computers ran to produce the impressive-looking output. Viewed from this perspective, Ragin (1987, 2000) is arguing skillfully and powerfully that many quantitative researchers are failing to do what they ought to be doing, and that insightful case studies are better than foolish regressions. He is correct, and I repeat his arguments to the graduate students every semester I teach statistics. He is also correct about the power of case study methods. I know of no other place where the same points are made more clearly and relevantly for the social sciences. For these two contributions, I cheer him. Where Professor Ragin divides from Seawright and me, however, is in what we want to do about this state of affairs. Faced with drunk driving, I do not want to abolish automobiles. Instead, I want to get the drunks off the road. Confronted with the divorce rate, I do not favor legislating against romance. I prefer to encourage more mature marriages. And reading all the uninspired quantitative applications in the social sciences, I do not seek the abandonment of conventional statistical theory

Achen

31

and techniques. I would rather let interested graduate students learn how to do case studies, and then for those who want to add quantitative tools, get them past the introductory one or two courses in statistical methods so that they can see the powerful exploratory and data analytic opportunities that modern computing makes possible. I believe that with deep substantive knowledge of their topic, skill with case studies, and modern quantitative machinery at their command, frankly, QCA will offer little that they do not already know. Few social scientists dispute the need to combine qualitative and quantitative methods and evidence in the profession. The question is how. As Ragin (1987: chap. 5; 2000: chap. 1), I (1982), and many other scholars have said, first-rate social science theorizing seems to integrate the two in ways we do not fully understand. For example, contemporary case-study methods are difficult to explicate in conventional statistical theory, and yet they are frequently quite powerful and successful in ways that no statistical methods could match. An important clue is that they often carry out an implicit comparison against known background relationships, most obviously in single-case studies (2000: 206). But what is the precise inferential logic of this step and why is it so successful? No one knows. Similar remarks apply when case-study analysts query tacit or unrecognized statistical presuppositions. In our inability to make sense of the alternate approach from our own methodological perspective, we often reject it, and thus collectively fall into two solitudes. What we are missing in social science methodology, it seems to me, is a theoretical grasp of how qualitative and quantitative methodologies go together. King et al. (1994) took the imperial view from the statistical side, arguing that good case studywork should look like econometric thinking. In effect, they asked, Why cant a woman be more like a man? Ragin reverses the tables, as he notes (2000: 14), constructing a quantitative methodology that mimics case study logic. He asks, Why cant a man be more like a woman? Just as Brady and Collier (2004) have critiqued King et al.s grand claims, so also Seawright (2005) has spoken against Ragins. But there is an alternate agenda that perhaps all of us could come to share. Acquiring the inferential insights that would let us see both types of methodology, like both genders, as genuinely distinct but legitimate and complementarythat, in my view, has not yet been done. Adcock and Collier (2001) take an important step forward, and others are at work as well. It seems to me a critically important goal for social science methodologists. Notes
1. See also Ragin (2000: 6466, 8081; 2005). 2. Berk (2004) lists many relevant texts in his excellent bibliography. 3. Units of analysis rather than number of observations, since, as Ragin (2000: 6869) and others have noted, even a single unit in a case study may yield a rich number of observations over time and across social events.

References
Achen, Christopher H. 1992. Breaking the Iron Triangle: Social Psychology, Demographic Variables and Linear Regression in Voting Research. Political Behavior 14, 3 (September 1992): 5211. . 2002. Toward a New Political Methodology: Microfoundations and ART. Annual Review of Political Science 5: 423450.

32

Studies in Comparative International Development / Spring 2005

. Lets Put Garbage-Can Regressions and Garbage-Can Probits Where They Belong. Under review, Conflict Management and Peace Science. Adcock, Robert, and David Collier. 2001. Measurement Validity. American Political Science Review 95, 3 (September): 529546. Berk, Richard A. 2004. Regression Analysis. Thousand Oaks, CA: Sage. Bishop, Yvonne M., Stephen E. Fienberg, and Paul W. Holland. 1977. Discrete Multivariate Analysis. Cambridge, Massachusetts: MIT Press. Brady, Henry E., and David Collier. 2004. Rethinking Social Inquiry. Lanham, MD: Rowman and Littlefield. Braumoeller, Bear F. 2003. Causal Complexity and the Study of Politics. Political Analysis 11: 209 233. Breiman, Leo, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. 1984. Classification and Regression Trees. Belmont, CA: Wadsworth. King, Gary, Robert O. Keohane, and Sidney Verba. 1994. Designing Social Inquiry. Princeton: Princeton University Press. Ragin, Charles C. 1987. The Comparative Method. Berkeley: University of California Press. . 2000. Fuzzy-Set Social Science. Chicago: University of Chicago Press. . 2005. Core Versus Tangential Assumptions in Comparative Research. Studies in Comparative International Development 40(1): 3338. Rosenberg, Morris. 1968. The Logic of Survey Analysis. New York: Basic Books. Seawright, Jason. 2005. Qualitative Comparative Analysis vis--vis Regression. Studies in Comparative International Development 40(1): 326.