By the end of today you can

• measure numeric predictions with RMSE, MAE, and R², against a baseline;
• explain why ordinary regression overfits with many predictors;
• fit and tune ridge, lasso, and elastic net regression;
• read which predictors the lasso keeps, and what that does not mean;
• explain how boosting builds a model from small trees, and tune Gradient Boosting;
• choose a model by cross-validation, and report the final test.

Many predictors, few cases

A questionnaire, several rounds of measurement, and a background survey can supply 50 predictors for 60 people.

Ordinary regression estimates a coefficient for each, fits its own data almost perfectly, and predicts new cases badly.

Correlated predictors, like questionnaire items, make it worse.

The question: final GPA from year one

The university’s advisers meet students at the end of year one.

The outcome, GPA in semester 4, is a number: a regression problem.

One row per student, wide

year_one = s12.pivot(
    index='student_id',
    columns='semester',
    values=['gpa', 'sleep_hours', 'study_hours', 'exercise_days',
            'caffeine_mg', 'supervisor_meetings', 'wellbeing']
)
year_one.columns = [f"{col[0]}_s{col[1]}" for col in year_one.columns]

Column names become gpa_s1, gpa_s2, sleep_hours_s1, etc.

Only students who stayed have a final GPA

gpa_data = (
    students
    .merge(questionnaire, on='student_id')
    .merge(year_one, on='student_id')
    .merge(final_gpa, on='student_id')
    .drop(columns=['student_id', 'supervisor_id', 'considering_dropout'])
)

merge(final_gpa, on='student_id') keeps students with a final GPA: 37 students who left are excluded.

The model predicts only for students who stay, and must not be used to judge the leavers.

Split, folds, and recipe

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=2026
)
gpa_folds = KFold(n_splits=10, shuffle=True, random_state=2026)

gpa_recipe = ColumnTransformer([
    ('num', Pipeline([
        ('impute', SimpleImputer(strategy='median')),
        ('scale', StandardScaler())
    ]), numeric_cols),
    ('cat', Pipeline([
        ('impute', SimpleImputer(strategy='most_frequent')),
        ('dummy', OneHotEncoder(drop='first', sparse_output=False, handle_unknown='ignore'))
    ]), categorical_cols)
])

After one-hot encoding categorical variables: 51 predictors.

Errors of a numeric prediction

Each error is actual minus predicted (\(y - \hat{y}\)).

Three summaries of the errors

from sklearn.metrics import root_mean_squared_error, mean_absolute_error, r2_score

MAE and RMSE are in GPA points, lower is better: “typically off by about 0.2”.

Always compare with a baseline

null_pipe = Pipeline([
    ('prep', gpa_recipe),
    ('model', DummyRegressor(strategy='mean'))
])

Predict the training mean for everyone: RMSE 0.32.

Any useful model must do clearly better.

Ordinary regression with every predictor

lm_pipe = Pipeline([
    ('prep', gpa_recipe),
    ('model', LinearRegression())
])

Cross-validated RMSE 0.220: much better than the baseline.

With 422 students and 51 predictors, it works reasonably. Theses often have 60.

The same model on 60 students

lm_small = Pipeline([('prep', gpa_recipe), ('model', LinearRegression())]).fit(X_small, y_small)

Excellent on its own data, worse than predicting the average on new data.

Multicollinearity

Six stress items, two semesters of GPA: many predictors are strongly correlated.

Regression struggles to divide credit between them.

Coefficients become large, unstable, even of opposite signs, cancelling out on training data but not on new data.

Regularisation penalises large coefficients

\[ \text{sum of squared errors} + \alpha \times \text{size of coefficients} \]

A little shrinkage costs almost no training fit and buys substantial stability. Predictors must be standardised.

Ridge, lasso, and elastic net

In scikit-learn: Ridge(alpha=...), Lasso(alpha=...), ElasticNet(alpha=..., l1_ratio=...).

A lasso on the same 60 students

lasso_small = Pipeline([
    ('prep', gpa_recipe),
    ('model', Lasso(alpha=0.03, random_state=2026))
]).fit(X_small, y_small)

The penalty kept the model from chasing noise.

Shrinkage, coefficient by coefficient

The lasso keeps 11 of 51 predictors and shrinks even those.

Coefficients that make no sense

With 60 students, ordinary regression fits chance patterns. Shrunken coefficients cannot chase one sample’s noise.

Tuning penalty \(\alpha\)

alphas = np.logspace(-4, 0, 20)
lasso_grid = GridSearchCV(
    Pipeline([('prep', gpa_recipe), ('model', Lasso(random_state=2026, max_iter=5000))]),
    param_grid={'model__alpha': alphas},
    cv=gpa_folds, scoring='neg_root_mean_squared_error'
).fit(X_train, y_train)

20 penalties on a log scale, for ridge, elastic net, and lasso.

Too little, just right, too much

At the far right, lasso zeros every coefficient: back to baseline. Best: \(\alpha \approx\) 0.013.

What the lasso keeps

best_lasso = lasso_grid.best_estimator_
best_coefs = best_lasso.named_steps['model'].coef_

Of 51 predictors, the lasso keeps 9. Year-one GPA dominates.

One SD higher semester 2 GPA: about 0.15 points higher final GPA.

The lasso path

As the penalty falls, predictors enter one by one. The two GPA measures enter first.

Selected is not the same as causal

Sleep affects GPA (Chapter 8), yet the lasso drops it.

Its effect is already visible in year-one GPA, so it adds little to the prediction.

Among correlated predictors the lasso keeps one, and which one can change between samples. Predict with the lasso; explain with Chapters 8 and 10.

Boosting

flowchart LR
    A["Start:<br>the average"] --> B["Fit a small tree<br>to the errors"] --> C["Add it,<br>scaled down"]
    C -->|repeat hundreds of times| B

Each tree learns what the model so far missed. The learning rate keeps each step small.

Many weak trees add up to a strong model.

Boosting with one predictor

One step, a staircase, then a curve that follows the data.

Forests and boosting

GradientBoostingRegressor fits trees sequentially to correct the ensemble’s residual errors.

Tuning Gradient Boosting

gb_grid = GridSearchCV(
    Pipeline([('prep', gpa_recipe), ('model', GradientBoostingRegressor(n_estimators=500, random_state=2026))]),
    param_grid={'model__learning_rate': [0.01, 0.03, 0.1], 'model__max_depth': [1, 2, 4]},
    cv=gpa_folds, scoring='neg_root_mean_squared_error'
).fit(X_train, y_train)

Best: depth 1, RMSE 0.211.

Depth-1 trees add separate effects with no interactions. Shallow winning means few interactions to find.

Comparing the models

Lasso and elastic net on top, Gradient Boosting and ridge close behind, all better than ordinary regression. The lasso is chosen for using only 9 predictors.

The final test

y_pred_test = final_lasso_model.predict(X_test)

Off by 0.16 GPA points on average, capturing 66% of the variation.

Predicted against actual

Predictions are less spread than reality: extreme students are pulled towards the average.

How much does year one add to past grades?

gpa_only_pipe = Pipeline([
    ('impute', SimpleImputer(strategy='mean')),
    ('model', LinearRegression())
]).fit(X_train[['gpa_s1', 'gpa_s2']], y_train)

Past grades are by far the best predictor. Sleep, stress, and support matter (Chapter 8), but their influence is already in the grades.

In your field: engineering

c_lm = LinearRegression().fit(cX_train, cy_train)
c_gb = GradientBoostingRegressor(
    n_estimators=500, learning_rate=0.05, max_depth=4, random_state=1
).fit(cX_train, cy_train)

Concrete strength from its recipe and age: curves and interactions.

Boosting cuts RMSE from 10.90 to 4.55 MPa. Flexible models win when the data has structure for them.

Practical lab: the Chapter 13 playground

Work through the playground exercises in your browser, with hints and solutions.

The browser version uses Pyodide to run scikit-learn, NumPy, and pandas right in your browser.

Practical exercises 1–3: errors and shrinkage

1. MAE and RMSE by hand; change one prediction by 1.0 and compare.
2. Repeat the 60-student experiment with 150 students.
3. Ridge against lasso coefficients: how many are exactly zero?

Practical exercises 4–6: design and communication

4. Scale scores instead of 22 items: does the lasso’s RMSE change?
5. Gradient Boosting with 100, 500, and 1000 trees at learning rate 0.03.
6. Explain an RMSE of 0.2 to academic advisers in two sentences.

Try this yourself

Take a prediction problem from your own field:

• count your cases and your predictors;
• name a baseline prediction;
• decide whether a lasso, a ridge, or boosting suits the structure;
• plan how you would report the final test.

Troubleshooting guide (Part 1)

Troubleshooting guide (Part 2)

Completion checklist

• predictors are known at prediction time;
• a baseline is reported;
• RMSE, MAE, and R² are reported;
• predictors are standardised before regularising;
• penalty \(\alpha\) and mixture are tuned by cross-validation;
• selected predictors are not described as causes;
• the test set is used once.

Misconceptions to leave behind (Part 1)

Misconceptions to leave behind (Part 2)

The chapter in one sentence

With many predictors, shrink the coefficients or build from small trees, compare against a baseline by cross-validation, and never read a selected predictor as a cause.

Next: Chapter 14

The next chapter revisits clustering:

• what counts as a group;
• Gaussian mixture models and soft membership;
• clusters as dense regions: DBSCAN;
• evaluating a clustering;
• reporting the choices behind the clusters.

Questions

How many predictors, and how many cases, would your model have?

What is the simplest prediction your model must beat?