3G Quantitative Methods for Causal Inference and Policy Evaluation

Course Content

In this course, students will learn the logic and methods of policy evaluation. We will focus on good research designs that answer important causal questions in public policy. In doing so, we will review the technical skills necessary to conduct credible empirical research such as field experiments, differences-in-differences, instrumental variables, and regression discontinuity designs. More importantly, we will practice the thinking necessary to develop and evaluate good research designs.

Class sessions will involve a combination of lecture and discussion. Some sessions will review a particular set of empirical methods in detail, others will involve the detailed discussion of a single paper, and others will involve a mix of lecture and discussion revolving around a substantive topic in policy evaluation. Students are expected to prepare for each class session and participate in the discussions.

Course Objectives

1. Students will be able to design advanced policy evaluations with quantitative methods. In particular, they will be able to correctly apply i) regression discontinuity designs, ii) instrumental variable designs, and iii) difference-in-difference designs.
2. Students should be able to critically evaluate the empirical soundness of existing policy evaluations.

Course Prerequisites

Students are expected to have taken at least one graduate level course in statistics. Only a basic familiarity with probability, hypothesis testing, and regression is assumed.
The course will conduct statistical analyses using the R programming language. No prior experience in computer coding is necessary. The Instructor and/or teaching assistant will hold special sessions to introduce R. Students will primarily be responsible for running already-produced code, on pre-assembled datasets, rather than writing their own code.

Readings

Angrist, Joshua D., and Jorn-Steffen Pischke. Mastering Metrics: The path from cause to effect. Princeton University Press, 2014. (this will be provided by ESS)

Thinking Clearly with Data: A Guide to Quantitative Reasoning and Analysis Ethan Bueno de Mesquita and Anthony Fowler, Princeton University Press, 9780691214368

We will read selected papers throughout the quarter. Links to these papers, or drafts of them, will be provided. One book which is suggested for several lectures is the following:

Gerber, Alan S., and Donald P. Green. 2012. Field Experiments: Design, Analysis, and Interpretation (FEDAI). New York: W.W. Norton.

Though a more specialized treatment, this book is a must-have for aspiring experimentalists.

Background knowledge required

Maths

Linear Regression = elementary

Statistics

OLS = elementary

 Day 1: The experimental ideal

Mastering ’Metrics Chapter 1

Paluck, Elizabeth Levy, Paul Lagunes, Donald P. Green, Lynn Vavreck, Limor Peer, and Robin Gomila. 2015. “Does Product Placement Change Television Viewers’ Social Behav- ior?.” PloS one 10, no. 9.

Supplemental Readings:

Campbell, Donald T. 1969. “Reforms as Experiments.” American Psychologist 24: 409-29.

Ashworth, Scott and Berry, Christopher R. and Bueno de Mesquita. Theory and Credibility. Chapters 2, 4.

Gerber, Alan S., and Donald P. Green. 2012. Field Experiments: Design, Analysis, and Interpretation (FEDAI). New York: W.W. Norton. Chapter 2.

Day 2: The two lives of covariates 

Mastering ’Metrics Chapter 2. 

Lenz, Gabriel, and Alexander Sahn. “Achieving statistical significance with covariates and without transparency.” University of California, Berkeley (2019). 

Supplemental Readings: 

Gerber, Alan S., and Donald P. Green. 2012. Field Experiments: Design, Analysis, and Interpretation (FEDAI). New York: W.W. Norton. Chapter 4. 

Craig Volden, Alan E. Wiseman, and Dana Wittmer. “When are Women More Effective Lawmakers than Men?” American Journal of Political Science 57, no. 2 (2013): 326-341. 

Day 3: “Natural” experiments 

Dunning, Thad. “Improving causal inference: Strengths and limitations of natural experi- ments.” Political Research Quarterly 61, no. 2 (2008): 282-293. 

Sekhon, Jasjeet S., and Rocio Titiunik. “When natural experiments are neither natural nor experiments.” American Political Science Review 106, no. 1 (2012): 35-57. 

Fowler, Anthony, and B. Pablo Montagnes. “College football, elections, and false-positive results in observational research.” Proceedings of the National Academy of Sciences 112, no. 45 (2015): 13800-13804. 

Supplemental Readings: 

Washington, Ebonya L. 2008. “Female Socialization: How Daughters Affect Their Legislator Fathers.” American Economic Review 98(1). 

Day 4: Instrumental variable designs 

Mastering ’Metrics Chapter 3 

Acemoglu, Daron, Simon Johnson, and James A. Robinson. 2002. “The Colonial Origins of Comparative Development: An Empirical Analysis.” American Economic Review. 91(5): 1369—1401. 

Supplemental Readings: 

Gerber, Alan S., and Donald P. Green. 2012. Field Experiments: Design, Analysis, and Interpretation (FEDAI). New York: W.W. Norton. Chapter 5. 

Day 5: Regression discontinuity designs 

Mastering ’Metrics Chapter 4 

Lee, David S., Enrico Moretti, and Matthew J. Butler. “Do voters affect or elect policies? Evidence from the US House.” The Quarterly Journal of Economics 119, no. 3 (2004): 807-859. 

Supplemental Readings: 

Eggers, Andrew C., Anthony Fowler, Jens Hainmueller, Andrew B. Hall, and James M. Snyder, Jr. 2015. “On the Validity of the Electoral Regression Discontinuity Design for Es- timating Electoral Effects: New Evidence from Over 40,000 Close Races.” American Journal of Political Science 59(1):259-274. 

Day 6: Advanced topics in regression discontinuity designs 

de la Cuesta, Brandon and Kosuke Imai. 2016. “Misunderstandings about the Regression Discontinuity Design in the Study of Close Elections.” Annual Review of Political Science. 19: 375-396. 

Gelman, Andrew, and Adam Zelizer. “Evidence on the deleterious impact of sustained use of polynomial regression on causal inference.” Research & Politics 2, no. 1 (2015): 2053168015569830. 

Supplemental Readings: 

Cattaneo, Matias D., Nicolás Idrobo, and Roc ́ıo Titiunik. 2017. A Practical Introduction to Regression Discontinuity Designs. 

Gelman, Andrew, and Guido Imbens. “Why high-order polynomials should not be used in regression discontinuity designs.” Journal of Business & Economic Statistics 37, no. 3 (2019): 447-456. 

Day 7: Differences-in-differences design 

Mastering ’Metrics Chapter 5 

Duflo, Esther. 2001. “Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment.” American Economic Review, 91(4): 795—813. 

Supplemental Readings: 

Fouirnaies, Alexander, and Hande Mutlu-Eren. “English bacon: Copartisan bias in inter- governmental grant allocation in England.” The Journal of Politics 77, no. 3 (2015): 805-817. 

Day 8: Interference 

Gerber, Alan S., and Donald P. Green. 2012. Field Experiments: Design, Analysis, and Interpretation (FEDAI). New York: W.W. Norton. Chapter 5. 

Zelizer, Adam. ”Is Position-Taking Contagious? Evidence of Cue-Taking from Two Field Experiments in a State Legislature.” American Political Science Review 113, no. 2 (2019): 340-352. 

Supplemental Readings: 

Bowers, Jake, Mark M. Fredrickson, and Costas Panagopoulos. “Reasoning about interfer- ence between units: A general framework.” Political Analysis 21, no. 1 (2013): 97-124. 

Day 9: Mechanisms 

Imai, Kosuke, Luke Keele, and Dustin Tingley. “A general approach to causal mediation analysis.” Psychological methods 15, no. 4 (2010): 309. 

Supplemental Readings: 

Imai, Kosuke. “Do get-out-the-vote calls reduce turnout? The importance of statistical methods for field experiments.” American Political Science Review 99, no. 2 (2005): 283- 300. 

Gerber, Alan S., and Donald P. Green. “Correction to Gerber and Green (2000), replication of disputed findings, and reply to Imai (2005).” American Political Science Review 99, no. 2 (2005): 301-313. 

Day 10: Class open for student suggestions