Designing Social Inquiry

Designing Social Inquiry: Scientific Inference in Qualitative Research (or KKV) is an influential 1994 book written by Gary King, Robert Keohane, and Sidney Verba that lays out guidelines for conducting qualitative research.^[1] The central thesis of the book is that qualitative and quantitative research share the same "logic of inference."^[2] The book primarily applies lessons from regression-oriented analysis to qualitative research, arguing that the same logics of causal inference can be used in both types of research.^[3][1]

The text is often referred to as KKV within social science disciplines. The book has been the subject of intense debate among social scientists.^[4][5][6] The 2004 book Rethinking Social Inquiry, edited by Henry E. Brady and David Collier, is an influential summary of responses to KKV.^[5]

History

Robert Keohane recounts the origins of KKV as follows,^[7]

Designing Social Inquiry was not generated by puzzles of world politics. Instead, it was the result of serendipity. Sid Verba and I were friends, and when I joined the Harvard Government Department in 1985, he said that we should teach a course together. I regarded this remark as a welcoming pleasantry, typical of Sid's grace and warmth. Three years later I became chair of the department and in my first year as chair was forced to listen to 24 job talks. Most of these talks were dead on arrival, since the speaker had made fundamental mistakes in research design. I complained to colleagues, including Sid, and Gary King. Gary said the three of us should teach a course on research design together... I agreed, and we taught the course the following year... After the semester was over, Gary said: “We should teach the course again. And this time, we should write a book on this subject.” The next year we met regularly for a bag lunch, discussing not only themes of the course but drafts that one of us—most often Gary, which is why his name appears first on the book—had produced.

Contents

The goal of the book is guide researchers in producing valid causal inferences in social science research.^[8] The book primarily applies lessons from regression-oriented analysis to qualitative research, arguing that the same logics of causal inference can be used in both types of research.^[3][1] The authors argue “whether we study many phenomena or few… the study will be improved if we collect data on as many observable implications of our theory as possible.”^[8] The authors note that case studies do not necessarily have to be N=1 or few N: a case study can include many observations within a case (many individuals and entities across many time periods).^[9] KKV criticize Harry H. Eckstein's notion of "crucial case studies", warning that a single observation makes it harder to estimate multiple causal effects, more likely that there is measurement error, and risks that an event in a single case was caused by random error.^[10]

According to the authors, a strong research design requires both qualitative and quantitative research, a research question that poses an important and real question that will contribute to the base of knowledge about this particular subject, and a comprehensive literature review from which hypotheses (theory-driven) are then drawn. Data that are collected should be operationalized so that other researchers could replicate the study and achieve similar results. For the same reason, the reasoning process behind the analysis needs to be explicit. While gathering data the researcher should consider the observable implications of the theory in an effort to explain as much of the data as possible. This is in addition to examining the causal mechanisms that connect one variable to another.

While qualitative methods cannot produce precise measurements of uncertainty about the conclusions (unlike quantitative methods), qualitative scholars should give indications about the uncertainty of their inferences. KKV argue that "the single most serious problem with qualitative research in political science is the pervasive failure to provide reasonable estimates of the uncertainty of the investigator’s inferences."^[11]

According to KKV, the rules for good causal theories are that they need to:

• be falsifiable
• have internal consistency (generate hypotheses that do not contradict each other)
• have variation (explanatory variables should be exogenous and dependent variables should be endogenous)
• have "concrete" concepts (concepts should be observable)
• have "leverage" (the theory should explain much by little).^[12]

KKV sees process-tracing and qualitative research as being "unable to yield strong causal inference" due to the fact that qualitative scholars would struggle with determining which of many intervening variables truly links the independent variable with a dependent variable. The primary problem is that qualitative research lacks a sufficient number of observations to properly estimate the effects of an independent variable. They write that the number of observations could be increased through various means, but that would simultaneously lead to another problem: that the number of variables would increase and thus reduce degrees of freedom.^[1]

In terms of case selection, KKV warn against "selecting on the dependent variable". For example, researchers cannot make valid causal inferences about wars outbreak by only looking at instances where war did happen (the researcher should also look at cases where war did not happen). There is methodological problem in selecting on the explanatory variable, however. They do warn about multicollinearity (choosing two or more explanatory variables that perfectly correlate with each other). They argue that random selection of cases is a valid case selection strategy in large-N research, but warn against it in small-N research.

KKV reject the notion of "quasi-experiments", arguing that either all the key causal variables can be controlled (an experiment) or not (a non-experiment).

Reception

In his 2010 review, James Mahoney writes that the field of social science methodology has "benefited from KKV even as it has also moved beyond it."^[1] Critics of KKV have characterized the book's claims as "often simplistic, misleading and inappropriate as a guide for designing social inquiry."^[1] Quantitative scholars such as Henry E. Brady, Larry M. Bartels and David A. Freedman have argued that KKV overstate the strengths of quantitative research vis-a-vis qualitative research.^[1] Henry Brady and David Collier argue that KKV exaggerate the ability of quantitative research to identify uncertainty.^[5] They also argue that KKV exaggerate the risks of conducting inductive research and forming hypotheses post hoc.^[5]

Numerous scholars disagree with KKV in their claims that qualitative research should integrate standards from quantitative research.^[5][6][13] There are different logics to the manner in which qualitative research is conducted and what qualitative scholars seek and can do with their data.^[13] Brady and Collier argue that KKV give insufficient attention to these divergent logics, as well as the intrinsic tradeoffs between different methodological goals.^[5] Gary Goertz and James Mahoney dispute that the main difference between qualitative and quantitative research is the size of N. Instead, a primary difference is that qualitative scholars tend to do within-case analyses whereas quantitative scholars almost by definition do cross-case analyses.^[13]

Mahoney writes that KKV ignore set theory and logic in terms of evaluating causal inference. Whereas regression-oriented analyses seek to estimate average effects of certain outcomes, qualitative research seeks to explain why cases have certain outcomes.^[1] Thus, causal inference is not strengthened by expanding the size of N, but rather by carefully choosing cases, whose testing can strengthen or weaken a theory.^[1] Mahoney and Gary Goertz make an analogy with a murder case: a single piece of smoking gun evidence can conclusively show whether a person committed a murder.^[13]

Mahoney also writes that KKV give insufficient attention to concept formation, which is an essential aspect of theory construction and measurement, and one of the important ways that qualitative research can play a key role.^[1]

Ronald Rogowski criticizes how KKV treat qualitative social science research. Rogowski argues that there is too much focus on hypothesis-testing and too much caution against using single observations. Rogowski argues that KKV promotes a form of qualitative social science that is overly focused on hypothesis-testing, and that this limits scholars' questions, cases and ambitions.^[14][15] John J. Mearsheimer and Stephen M. Walt argue that International Relations scholarship has shifted away from crafting and refining IR theory to "simplistic hypothesis-testing", in part due to the influence of KKV in political science graduate programs.^[16]

Alexander George and Andrew Bennett say there is "much to agree with" in KKV, but they argue that the book has several flaws in its guidance on qualitative research:^[6]

• Causal mechanisms: KKV suggest that "causal mechanisms" are less important that "causal effects" in causal explanations – George and Bennett argue that they are equally important
• Hypothesis-testing: KKV overly emphasize the role of hypothesis-testing in theory development – George and Bennett argue that the formation of new hypotheses and the choosing of new questions are also important aspects of theory development
• Causal complexity: KKV fail to consider problems of causal complexity, such as equifinality, multiple interaction effects, feedback loops, path dependency, tipping points, selection effects, expectations effects and sequential interactions – George and Bennet argue that case studies, process tracing and typological theories can clarify causality in situations of causal complexity
• Increasing N: KKV argue that scholars should always seek to increase the number of cases – George and Bennett argue that KKV fail to consider the costs to increasing the number of cases (such as conceptual stretching and unintentional comparisons of dissimilar cases). George and Bennett note that much value can be derived from single-case studies.
• Process-tracing: KKV characterize process-tracing as a way to increase the number of observable implications – George and Bennett argue that the logic of process-tracing is entirely different. The logic behind using process-tracing is to focus on the sequences and timings within a case, not to correlate data across cases. Thus, if one piece of evidence in the sequence is inconsistent with the theoretical expectations, then the theory has been shown to be flawed.
• "Degrees of freedom" problem: KKV argue that a single case cannot evaluate competing explanations due problems that arise from degrees of freedom – George and Bennett argue that it is flawed to apply this statistical logic to qualitative research. George and Bennett say that while quantitative scholars try to aggregate variables to reduce the number of variables and thus increase the degrees of freedom, qualitative scholars intentionally want their variables to have many different attributes and complexity.

Further reading

• Review symposium in the American Political Science Review Vol. 89, No. 2, Jun., 1995
• Gary Goertz and James Mahoney provide a bullet point summary of KKV's contents in their 2012 book, A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences.

References

1. Mahoney, James (2010). "After KKV: The New Methodology of Qualitative Research". World Politics. 62 (1): 120–147. doi:10.1017/S0043887109990220. ISSN 1086-3338. S2CID 43923978.
2. King, Gary/ Keohane, Robert O./ Verba, Sidney: Designing Social Inquiry. Scientific Inference in Qualitative Research, p. 3. Princeton University Press, 1994.
3. Humphreys, Macartan; Jacobs, Alan M. (2015). "Mixing Methods: A Bayesian Approach". American Political Science Review. 109 (4): 654. doi:10.1017/s0003055415000453. ISSN 0003-0554. S2CID 1846974.
4. Morgan, Kimberly J. (2016-09-02). "Process tracing and the causal identification revolution". New Political Economy. 21 (5): 489–492. doi:10.1080/13563467.2016.1201804. ISSN 1356-3467. S2CID 156377366.
5. Brady, Henry E. Collier, David (2010). Rethinking social inquiry : diverse tools, shared standards (2 ed.). Rowman & Littlefield Publishers. ISBN 978-1-4422-0343-3. OCLC 838295613.{{cite book}}: CS1 maint: multiple names: authors list (link)
6. George, Alexander L.; Bennett, Andrew (2005). Case Studies and Theory Development in the Social Sciences. MIT Press. pp. 10–18, 28–33. ISBN 978-0-262-30307-1. OCLC 944521872.
7. Keohane, Robert O. (2020-05-11). "Understanding Multilateral Institutions in Easy and Hard Times". Annual Review of Political Science. 23 (1): 1–18. doi:10.1146/annurev-polisci-050918-042625. ISSN 1094-2939.
8. King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 1–4, 12. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
9. King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 52–53. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
10. King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 209–211. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
11. King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. p. 32. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
12. King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 99–114. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
13. Goertz, Gary; Mahoney, James (2012-09-09). A Tale of Two Cultures. Princeton University Press. pp. 1–2, 10. doi:10.23943/princeton/9780691149707.001.0001. ISBN 978-0-691-14970-7.
14. Rogowski, Ronald (2010). "How Inference in the Social (but Not the Physical) Sciences Neglects Theoretical Anomaly" in Rethinking social inquiry diverse tools, shared standards. Rowman & Littlefield Publishers. ISBN 978-1-4422-0343-3. OCLC 787870333.
15. Rogowski, Ronald (1995). "The Role of Theory and Anomaly in Social-Scientific Inference". American Political Science Review. 89 (2): 467–470. doi:10.2307/2082443. ISSN 1537-5943. JSTOR 2082443. S2CID 146265258.
16. Mearsheimer, John J.; Walt, Stephen M. (2013). "Leaving theory behind: Why simplistic hypothesis testing is bad for International Relations". European Journal of International Relations. 19 (3): 427–457. doi:10.1177/1354066113494320. ISSN 1354-0661. S2CID 52247884.