*We are no longer accepting applications for this course*
Arnaud Vaganay is a methodologist and meta-researcher. He currently heads Meta-Lab, a London-based organisation specialised in meta-research (or research-on-research) and open science training. He is also a visiting lecturer at the London School of Economics and Sciences Po. His academic research meta-analyses the effect of policy commitments, research sponsorship and skills on the transparency and credibility of social policy evaluations. His publications cover topics including sampling bias, reporting bias, sponsorship bias and time preferences for evidence.
Dr Thomas J. Leeper. Thomas J. Leeper is an Assistant Professor in Political Behaviour in the Department of Government at the London School of Economics and Political Science. His research, which primarily focuses on the role of information in politics, has been published in American Political Science Review, American Journal of Political Science, Public Opinion Quarterly, and other journals. He has developed over 30 published R software packages as part of the rOpenSci, rOpenGov, and cloudyr development projects. He received his PhD from Northwestern University and was previously a postdoc at Aarhus University.
Reproducibility is the ability of an entire study to be duplicated, either by the same researcher or by someone else working independently. As such, reproducibility is one of the main principles of the scientific method.
Although most researchers are committed to the principle of reproducibility, few actually achieve it. By some accounts, only half of studies published in social science journals are reproducible.
This course offers a set of methods to make research more traceable, for the benefit of both:
Authors, through more efficient and better documented workflows; and
Research users, including editors, citing authors, knowledge-brokers, through a better understanding of what the authors did and why they did it.
The course follows the research cycle through four key stages:
Data collection and analysis;
At each stage, participants will:
Discuss the ‘gold standard’ of reproducible research;
Discuss the main risks and obstacles to reproducible research;
Engage with applied examples of open (and less open) empirical studies;
Test different apps and software tools such as Git, Knitr, OSF and Dataverse with the aim of streamlining their own workflow.
Examples will be drawn from across the social sciences and students will have the opportunity to work their preferred statistical software (with a strong preference shown for R or Stata).
Upon completion of the course, students will be able to:
1. Recall and discuss the causes and consequences of irreproducible research;
2. Assess the reproducibility of a given empirical study;
3. Implement transparent and reproducible practices in their own workflows;
4. Apply these skills through the use of open science software and apps.
Our course has been developed to address the specific needs of researchers:
With a completed BSc (minimum);
With no or limited prior experience of open science tools/methods;
Committed to adopt these tools/methods soon after the activity;
With demonstrated experience in conducting empirical research and analysing quantitative data;
With a good grasp of the social science literature (economics, political science, psychology, sociology, social work, etc.) or the public health literature;
Reasonably familiar with Stata or R.
Representative Background Reading
Here are some studies illustrating the issue of irreproducibility in a few disciplines:
Ioannidis JP (2005) Why most published research findings are false. PLoS Med. 2(8): e124. doi:10.1371/journal.pmed.0020124
Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science, 28 Aug 2015: Vol. 349, Issue 6251, DOI: 10.1126/science.aac4716
Bailey DH, Borwein JM, Lopez de Prado M, Qiji JZ (2014) Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society, 61(5), May 2014, pp.458-471.
Chang AC, Li P (2015) Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ”Usually Not”. Finance and Economics Discussion Series 2015-083. Washington: Board of Governors of the Federal Reserve System, http://dx.doi.org/10.17016/FEDS.2015.083.
In political science:
Esarey J, Wu A (2016) Measuring the effects of publication bias in political science. Research & Politics 3(3). https://doi.org/10.1177/2053168016665856
In health/medical research:
Iqbal SA, Wallach JD, Khoury MJ, Schully SD, Ioannidis JPA (2016) Reproducible Research Practices and Transparency across the Biomedical Literature. PLoS Biol 14(1): e1002333. doi:10.1371/journal.pbio.1002333
Begley CG, Ellis LM (2012) Drug development: Raise standards for preclinical cancer research. Nature 483: 531–533.
Prinz F, Schlange T, Asadullah K (2011) Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov 10: 712.
In social policy research:
Vaganay A (2016) Outcome Reporting Bias in Government-Sponsored Policy Evaluations: A Qualitative Content Analysis of 13 Studies. PLoS ONE 11(9): e0163702. doi:10.1371/journal.pone.0163702
Suggested texts include:
Manual of Best Practices in Transparent Social Science Research, by Garret Christensen (BITSS, 2016). Available at: http://www.bitss.org/education/manual-of-best-practices/
The Workflow of Data Analysis Using Stata by J. Scott Long (Stata Press, 2008)
Reproducible Research with R & RStudio by Christopher Gandrud (Chapman & Hall/CRC, 2013)
Implementing Reproducible Research edited by Victoria Stodden, Friedrich Leisch, and Roger D. Peng (Chapman & Hall/CRC, 2014).
The Practice of Reproducible Research edited by Justin Kitzes, Daniel Turek, and Fatma Imamoglu (under review at Oxford and UC Press).