1J Measurement: From simple classification to model estimation

Please note: This course will be taught in hybrid mode. Hybrid delivery of courses will include synchronous live sessions during which on campus and online students will be taught simultaneously.

Course Objectives
The purpose of the course is to give students an introduction on the strategies to achieve systematic, flexible and appropriate measurement indicators for concepts of interest. The set of tools includes Bayesian inference modelling for combining variables and estimate latent traits, and the systematic use of Large Generative Artificial Intelligence Models (LGAIMs, also known as LLMs) to experiment with measures for textual data. It aims to allow the development of strategies for transforming raw data into meaningful measures of concepts in the social sciences.
Developing good measures of the concepts to work with is the first step for assessing causal relationships and associations amongst multiple variables.  At the end of the course participants
will be able to design a system to transform raw measures into meaningful variables; create clusters and typologies of observations; and perform measurement of such concepts through time. They will also be able to employ Large Generative AI Models (LLMs) hand-in-hand with inference based Bayesian methods to produce latent variables of interest, specially in the context of observational data collection.

Course Prerequisites
Some familiarity with the linear model and/or with logit/probit regression is helpful, but not essential. No specific software requirements are necessary, since only Free and Open Source Software, as well as open weight LLMs will be used. R, JAGS and software for managing Large Language Models will be covered from the beginning.

Background Knowledge
Statistics:
OLS – elementary
Maximum Likelihood – elementary

Software:
R – moderate

Maths:
Linear Regression – elementary
Calculus – elementary

The course has 10 sessions of 2 lecture hours and 1.5 lab hours each day. Labs give students the possibility to work with the datasets provided by the instructor or with their own datasets.

1. Introduction
The first session reviews challenges, opportunities and limitations of measurement, as well as notation and fundamental prerequisites (such as probability distributions) used later in the course. A basic introduction to Bayesian inference is also performed.

2. Continuous variables: factor analysis
One session covers the fundamentals of factor analysis for continuous variables. The Human Development Index from the UNDP is presented and discussed.

3. Binary variables: item–response models
One session covers item–response models for binary indicators. This includes the Rasch one-parameter model and the two and three logistic parameter models [4] Besides the
uses, the session also reviews the limitations of such models. The examples examined include the following: generating an indicator out of several binary survey questions; ideal points of US legislators; and a measure of the complexity of European Regulatory Agencies [8].

4. Ordered variables and Mixed measurement (continuous, binary and ordinal)
This session covers the transformation of ordinal variables into continuous, by generating data-driven cutting points in each category. An application on how to measure democracy [17] is examined. The session also extends the continuous, binary and ordinal models into mixed factor analysis, that combines both indicators and also ordered variables. Relying on Quinn [13], an example about the degree of formalism of legal systems [15] is provided, as well as an example on the degree of independence and accountability of regulatory agencies.

5. Missing data as latent data, and extensions: Temporal dynamics and Mixed sources
One session specifically deals with the advantages of Bayesian inference for dealing with missing data and how this approach can be seen as a way to model latent data. How to extend measurement models to account for temporal variability is revised, making special emphasis on simple time-series models and the use of Kalman filters. Examples on morality policies and regional autonomy, federalism and decentralization are presented. How to mix data from several sources and aggregate it in order to obtain pooled indicators of the desired measures. An example about pooling data from different polling sources is revised [10]. Also, a discussion about the aggregation performed at the Worldwide Governance Indicators [18] is performed.

6. Classification, clustering and typologies
One session provides an introduction to clustering and classification. By relying on [5], an example about classification of welfare regimes in Western countries [1] is revised. Other examples include classifying European drug regimes, institutional features of organizations [11] and typologies of regions.

7. Conjoint analysis
This session provides a concrete example of the use of Hierarchical Bayes (HB) approaches to conjoint analysis, extracting every surveyed individual with the preference for each
of the profiles. A revision of [7] will be covered, with revised readings from [2] and [16].

8. Text analysis as a measurement challenge
A final set of 3 sessions session introduce text analysis as part of a more general effort in producing valid, reliable and systematic measures of large streams of documents. It relies on employing Large Generative Artificial Models (LGAIMs, also known as LLMs) to experiment with codification. The sessions provide specific tools on how to effectively employ LGAIMs / LLMs to be later on combined with proper measurement models.