Please note: This course will be taught online only. In person study is not available for this course. 

Reto Wüest is a postdoctoral researcher in the Department of Comparative Politics at the University of Bergen. He holds a PhD from the University of Geneva. Reto’s methodological research focuses on measurement and the leveraging of machine learning techniques to improve the prediction accuracy of multilevel regression and post-stratification models. His substantive research focuses on political representation and legislative politics in Europe and the US. Reto has been teaching machine learning courses at the Essex Summer School in Social Science Data Analysis, the Barcelona Summer School in Survey Methodology, the University of Geneva, and the University of Bergen. His work has been published in Journal of Politics, European Journal of Political Research, and West European Politics, among others.

Course content

Machine learning refers to the automated detection of meaningful patterns in data. Not only are machine learning techniques ubiquitous today in that they are behind many of the technologies we use in our daily lives, but they have also become an important part of the social scientist’s toolkit, especially since more and more social science data are now available in electronic form. A common feature of all applications of machine learning is the use of computer algorithms that can “learn” and adapt. The goal of this course is to introduce participants to a variety of widely used supervised and unsupervised machine learning methods. After discussing the fundamental concepts of machine learning, the course continues with supervised learning methods (e.g., regularization, tree boosting, support vector machines, and neural networks), then turns to unsupervised learning methods (e.g., principal components analysis and clustering methods), and lastly covers flexible model averaging techniques (e.g., Bayesian model averaging).

Course objectives

Participants will gain a solid understanding of a number of widely used and powerful machine learning methods. By the end of the course, they should be able to apply all the methods covered in class using the R statistical computing environment. They should also be able to explain the logic underlying the different methods, to understand the respective strengths and weaknesses of these methods, and to interpret their results.

Course prerequisites

Participants should have a solid understanding of probability theory and regression analysis. They should also be familiar with basic programming in R.

Course text 

James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2021. An Introduction to Statistical Learning with Applications in R. 2nd ed. New York: Springer. Available at https://www.statlearning.com/

Background knowledge/skills required:

OLS: Moderate

Maximum Likelihood: Elementary

R: Moderate

Maths:

Calculus: Elementary

Linear Regression: Moderate

Course Outline

Week 1

Day 1 – Statistical Learning; Resampling Methods. Goals and challenges of machine learning; supervised and unsupervised learning; regression and classification problems; bias-variance trade-off and the problem of overfitting; model assessment and selection; cross-validation (k-fold cross-validation, leave-one-out cross-validation); bootstrapping.

Literature:

James et al. 2021. Chapter 2 and Chapter 5.

Day 2 – Linear Methods for Regression. Linear regression and least squares; subset selection (best subset selection, stepwise selection); shrinkage methods (ridge regression, lasso, elastic net).

Literature:

James et al. 2021. Chapter 3 and Chapter 6 (6.1-6.2, 6.5.1-6.5.2).

Day 3 – Linear (and Other) Methods for Classification. Logistic regression; linear discriminant analysis, k-nearest neighbors.

Literature:

James et al. 2021. Chapter 4 (4.1-4.5, 4.7.1-4.7.6).

Day 4 – Moving Beyond Linearity. Polynomial regression; piecewise-constant regression; regression and smoothing splines; local regression; generalized additive models.

Literature:

James et al. 2021. Chapter 7.

Day 5 – Tree-Based Methods. Single decision trees; tree pruning; bagging; random forests; tree boosting; Bayesian additive regression trees.

Literature:

James et al. 2021. Chapter 8.

Week 2

Day 6 – Support Vector Machines. Maximal margin classifier; support vector classifier; support vector machine.

Literature:

James et al. 2021. Chapter 9.

Day 7 – Deep Learning. Single layer neural networks; multilayer neural networks; convolutional neural networks; recurrent neural networks.

Literature:

James et al. 2021. Chapter 10.

Day 8 – Principal Components Analysis. Principal components analysis; interpretation of principal components; principal components regression.

Literature:

James et al. 2021. Chapter 12 (12.1-12.2, 12.5.1) and Chapter 6 (6.3, 6.5.3).

Day 9 – Clustering Methods. K-means clustering; k-medoids clustering; hierarchical clustering.

Literature:

James et al. 2021. Chapter 12 (12.4, 12.5.3).

Day 10 – Model Averaging; MrP, autoMrP. Ensemble Bayesian model averaging; multilevel regression and poststratification (MrP); MrP and machine learning (autoMrP).

Literature:

Montgomery, Jacob M., Florian M. Hollenbach and Michael D. Ward. 2012. “Improving Predictions using Ensemble Bayesian Model Averaging.” Political Analysis 20(3): 271-291.

Broniecki, Philipp, Lucas Leemann, and Reto Wüest. 2022. “Improved Multilevel Regression with Poststratification through Machine Learning (autoMrP).” Journal of Politics 84(1): 597-601.