3L Ideal Point Estimation, Item Response Theory, and Scaling Methods for Surveys and Behaviour

Course Content

This course focuses on methods to discover, understand and visualize latent patterns in data and is especially suited to students with projects using survey data and other forms of relational data used in political science, sociology, economics, business, marketing, and psychology. The course introduces students to measurement theory and methods of scaling techniques, integrating Multidimensional Scaling, Item Response Theory, and Ideal Point Estimation. The first part of the course will provide an overview of the foundations of these techniques and introduce students to the most common methods for scaling and “spatial” analysis and the visualization of latent patterns in survey and behaviour data. Key themes of the course include:

• How can we estimate preferences, traits, and other difficult-to-measure concepts?
• How can we improve comparability of latent concepts across survey respondents, or across separate surveys?
• How can we reduce complex behavioural data to its basic underlying dimensions?

The course will demonstrate how to interpret, measure, and visualize latent dimensions of data via a variety of scaling methods using the open-source programming language R. The course will also discuss a range of applications these methods to social science studies of relational and perception data derived from elite behaviour and surveys, especially for identifying latent preferences of political, economic and social actors. The course concludes with discussions of the most recent advances in the field, including applications for text analysis, and practical advice for those seeking to use such methods in social science research, relevant to the students enrolled.

The course first covers how to analyse data from scales found in surveys (such as Likert-type scales), focusing on surveys that ask respondents to place themselves and / or stimuli on issue or attribute scales. The course begins with approaches to scaling to generate bias-adjusted and latent spatial data from survey responses, such as the Aldrich-McKelvey scaling and ‘Basic Space’ scaling with Anchoring Vignettes as methods for addressing perceptual bias in the form of “Differential Item Functioning.”  The course next examines similarities data and rating scale data from surveys (such as favourability scales for stimuli such as politicians or social groups) using multidimensional scaling (MDS), such as the SMACOF optimization method implemented in R and Bayesian applications to Metric Multidimensional Scaling. Finally, the course provides an overview of Item Response Theory and ideal point estimation, generally focused on binary choice data, which includes those used in ‘roll call voting’ analysis of elite behaviour in parliaments and courts. Here we will cover Poole and Rosenthal’s W-NOMINATE and Poole’s Optimal Classification unfolding method, as well as a variety of Bayesian analysis techniques for binary and ordinal choice data using Item Response Theory (IRT). An extensive range of Bayesian techniques is discussed, including Bayesian Aldrich-McKelvey Scaling, Ordinal and Dynamic Item Response Theory, Bayesian Multidimensional Scaling (MDS), and Bayesian Unfolding. The final section will discuss recent methods for scaling to a variety of different data types, including text data and the latest computation innovations to apply scaling methods to large-scale data sets.

This course will enable students to derive latent spatial preference information and/or a dimensional structure from various types of survey and behaviour data, which is applicable to a wide range of social science applications, academic and non-academic alike. Consumers of research based on these methods will also benefit from a deeper understanding of this type of methodology, its potential and its limitations.

Data types covered for scaling, IRT and/or ideal point applications include:

• Survey Response Data, including issue opinions, placements and Likert Scales
• Perceptual, Preferential, and Similarities Data
• Elite behaviour and choice data (e.g., ‘roll call’ voting in legislatures and courts)
• Text Data

Course Objectives

Students will learn to use various computational methods to generate measures of ideology and preferences and understand the latent dimensional properties of social science data, including surveys and legislative data. Students will understand the theories behind these methods and the relationships between Item Response Theory, Ideal Point Estimation and other scaling methods. As these techniques are fundamental parts of much recent work in social science, students will be able to both understand and produce this research based on measuring concepts in this way. In addition, students will be able to:

• Understand the fundamentals of measurement theory and scaling techniques,
• Explain the advantages and limitations of various scaling techniques
• Demonstrate proficiency in using relevant software packages in R
• Identify appropriate methods of measurement for a given set of data.
• Analyse the results of measurement techniques and present them visually.

Course Prerequisites

The course is designed to be accessible to social science graduate students of all backgrounds. However, students familiar with the R programming environment will find it easier to adapt to course content and assignments, so it is recommended to become familiar with the basic structure of the R syntax and the Rstudio software prior to enrolling. The 1-day Introduction to R offered the Sunday before the first day of class at ESS is recommended for this purpose. In addition, the course assumes some basic familiarity with general statistics (OLS and MLE).

Reading:

Armstrong, D. A., Bakker, R., Carroll, R., Hare, C., Poole, K. T., & Rosenthal, H. (2020). Analyzing Spatial Models of Choice and Judgment (2nd ed). : CRC Press ISBN 9780367612542 – (this text will be provided by ESS).

Background knowledge required

Statistics

OLS = elementary

Maximum Likelihood = elementary

Computer Background

R = elementary

Introduction: Overview of Scaling Methods, IRT, and Ideal Point Estimation – Core reading Ch 1 of Armstrong et al. Discussion of scaling methods in general and the general theory behind Ideal point estimation models. 

Scaling Single-item scales, identifying complexity and differential item functioning – Core reading Ch 2 of Armstrong et al.
Supplementary reading Zakharova, M. and Warwick, P.V., 2014. The Sources of Valence Judgments: The Role of Policy Distance and the Structure of the Left–Right Spectrum. Comparative Political Studies, 47(14), pp.2000-2025.

Scaling Issue Multiple Issue Scales –
Core Reading Ch 2 of Armstrong et al.
Supplementary reading Bakker, R., Jolly, S., Polk, J. and Poole, K., 2014. The European common space: Extending the use of anchoring vignettes. The Journal of Politics, 76(4), pp.1089-1101.

Similarities data, Agreement scores and Rating scale data 
Core reading Ch 3 and 4 of Armstrong et al.
Supplementary reading  Bornschier, S., 2010. The new cultural divide and the two-dimensional political space in Western Europe. West European Politics, 33(3), pp.419-444

Binary data and NOMINATE Ideal Point Estimation and Non-parametric Unfolding with Optimal Classification 
Core reading Ch 5 of Armstrong et al.
Supplementary reading Hix, Simon, Abdul Noury, and Gérard Roland. “Voting patterns and alliance formation in the European Parliament.” Philosophical Transactions of the Royal Society of London B: Biological Sciences 364.1518 (2009): 821-831.
Rosenthal, Howard, and Erik Voeten. “Analyzing roll calls with perfect spatial voting: France 1946–1958.” American Journal of Political Science 48.3 (2004): 620-632.

Item Response Theory and applications to binary choice data
Core reading Ch 5 of Armstrong et al.
Clinton, J., Jackman, S. and Rivers, D., 2004. The statistical analysis of roll call data. American Political Science Review, 98(2), pp.355-370.

Additional IRT models: Ordinal data and dynamic IRT
Core reading Ch 6 of Armstrong et al

Scaling for Text Data and other ‘big data’ sources
Core reading Ch 6 of Armstrong et al  and  Imai, K., Lo, J. and Olmsted, J., 2016. Fast estimation of ideal points with massive data. American Political Science Review, 110(4), pp.631-656.
Supplementary reading Barberá, Pablo. “Birds of the same feather tweet together: Bayesian ideal point estimation using Twitter data.” Political Analysis 23.1 (2014): 76-91. Martin, Andrew D., and Kevin M. Quinn. “Dynamic ideal point estimation via Markov chain Monte Carlo for the US Supreme Court, 1953–1999.” Political Analysis 10.2 (2002): 134-153. Slapin, Jonathan B., and Sven‐Oliver Proksch. “A scaling model for estimating time‐series party positions from texts.” American Journal of Political Science 52.3 (2008): 705-722