By the end of today you can

• measure numeric predictions with RMSE, MAE, and R², against a baseline;
• explain why ordinary regression overfits with many predictors;
• fit and tune ridge, lasso, and elastic net regression;
• read which predictors the lasso keeps, and what that does not mean;
• explain how boosting builds a model from small trees, and tune XGBoost;
• choose a model by cross-validation, and report the final test.

Many predictors, few cases

A questionnaire, several rounds of measurement, and a background survey can supply 50 predictors for 60 people.

Ordinary regression estimates a coefficient for each, fits its own data almost perfectly, and predicts new cases badly.

Correlated predictors, like questionnaire items, make it worse.

The question: final GPA from year one

The university’s advisers meet students at the end of year one.

The outcome, GPA in semester 4, is a number: a regression problem.

One row per student, wide

year_one <- semesters |>
  filter(semester <= 2) |>
  pivot_wider(id_cols = student_id, names_from = semester,
              values_from = gpa:wellbeing,
              names_glue = "{.value}_s{semester}")

names_glue builds names such as gpa_s1 and sleep_hours_s2.

Only students who stayed have a final GPA

gpa_data <- students |>
  left_join(questionnaire, join_by(student_id)) |>
  left_join(year_one, join_by(student_id)) |>
  inner_join(final_gpa, join_by(student_id)) |>
  select(-student_id, -supervisor_id, -considering_dropout)

inner_join() keeps students with a final GPA: 37 who left are excluded.

The model predicts only for students who stay, and must not be used to judge the leavers.

Split, folds, and recipe

gpa_split <- initial_split(gpa_data, prop = 0.75, strata = final_gpa)
gpa_folds <- vfold_cv(gpa_train, v = 10, strata = final_gpa)

gpa_recipe <- recipe(final_gpa ~ ., data = gpa_train) |>
  step_impute_median(all_numeric_predictors()) |>
  step_impute_mode(all_nominal_predictors()) |>
  step_dummy(all_nominal_predictors()) |>
  step_normalize(all_numeric_predictors())

With a numeric outcome, strata samples within quartiles. After dummies: 51 predictors.

Errors of a numeric prediction

Each error is actual minus predicted.

Three summaries of the errors

regression_metrics <- metric_set(rmse, mae, rsq)

MAE and RMSE are in GPA points, lower is better: “typically off by about 0.2”.

Always compare with a baseline

null_wf <- workflow(gpa_recipe, null_model(mode = "regression"))
null_cv <- fit_resamples(null_wf, resamples = gpa_folds,
                         metrics = metric_set(rmse, mae))

Predict the training mean for everyone: RMSE 0.32.

Any useful model must do clearly better.

Ordinary regression with every predictor

lm_wf <- workflow(gpa_recipe, linear_reg())
lm_cv <- fit_resamples(lm_wf, resamples = gpa_folds,
                       metrics = regression_metrics)

Cross-validated RMSE 0.218: much better than the baseline.

With 422 students and 51 predictors, it works reasonably. Theses often have 60.

The same model on 60 students

set.seed(3)
small_train <- gpa_train |> slice_sample(n = 60)
lm_small <- fit(lm_wf, data = small_train)

Excellent on its own data, worse than predicting the average on new data.

Multicollinearity

Six stress items, two semesters of GPA: many predictors are strongly correlated.

Regression struggles to divide credit between them.

Coefficients become large, unstable, even of opposite signs, cancelling out on training data but not on new data.

Regularisation penalises large coefficients

\[ \text{sum of squared errors} + \lambda \times \text{size of the coefficients} \]

A little shrinkage costs almost no training fit and buys much stability. Predictors must be normalised.

Ridge, lasso, and elastic net

In tidymodels: linear_reg(penalty = , mixture = ) with the glmnet engine.

A lasso on the same 60 students

lasso_small <- workflow(gpa_recipe,
                        linear_reg(penalty = 0.03, mixture = 1) |>
                          set_engine("glmnet")) |>
  fit(data = small_train)

The penalty kept the model from chasing noise.

Shrinkage, coefficient by coefficient

The lasso keeps 12 of 51 predictors and shrinks even those.

Coefficients that make no sense

With 60 students, ordinary regression fits chance patterns. Shrunken coefficients cannot chase one sample’s noise.

Tuning penalty and mixture

glmnet_wf <- workflow(gpa_recipe,
  linear_reg(penalty = tune(), mixture = tune()) |> set_engine("glmnet"))

glmnet_grid <- expand_grid(penalty = 10^seq(-4, 0, length.out = 20),
                           mixture = c(0, 0.5, 1))

glmnet_tuning <- tune_grid(glmnet_wf, resamples = gpa_folds,
                           grid = glmnet_grid, metrics = metric_set(rmse))

20 penalties on a log scale, for ridge, an even elastic net, and lasso.

Too little, just right, too much

At the far right the lasso has zeroed every coefficient: back to the baseline. Best: mixture 1, penalty 0.013.

What the lasso keeps

best_penalty <- select_best(glmnet_tuning, metric = "rmse")
lasso_fit <- glmnet_wf |> finalize_workflow(best_penalty) |> fit(data = gpa_train)
tidy(lasso_fit) |> filter(estimate != 0) |> arrange(desc(abs(estimate)))

Of 51 predictors, the lasso keeps 10. Year-one GPA dominates.

One SD higher semester 2 GPA: about 0.15 points higher final GPA.

The lasso path

As the penalty falls, predictors enter one by one. The two GPA measures enter first.

Selected is not the same as causal

Sleep affects GPA (Chapter 8), yet the lasso drops it.

Its effect is already visible in year-one GPA, so it adds little to the prediction.

Among correlated predictors the lasso keeps one, and which one can change between samples. Predict with the lasso; explain with Chapters 8 and 10.

Boosting

flowchart LR
    A["Start:<br>the average"] --> B["Fit a small tree<br>to the errors"] --> C["Add it,<br>scaled down"]
    C -->|repeat hundreds of times| B

Each tree learns what the model so far missed. The learning rate keeps each step small.

Many weak trees add up to a strong model.

Boosting with one predictor

One step, a staircase, then a curve that follows the data.

Forests and boosting

XGBoost is a fast, popular implementation, among the most successful methods for tables of data.

Tuning XGBoost

xgb_wf <- workflow(gpa_recipe,
  boost_tree(trees = 500, learn_rate = tune(), tree_depth = tune()) |>
    set_engine("xgboost") |> set_mode("regression"))

tune_grid(xgb_wf, resamples = gpa_folds,
          grid = expand_grid(learn_rate = c(0.01, 0.03, 0.1),
                             tree_depth = c(1, 2, 4)))

Best: depth 1, RMSE 0.212.

Depth-1 trees add separate effects with no interactions. Shallow winning means few interactions to find.

Comparing the models

Lasso and elastic net on top, XGBoost and ridge close behind, all better than ordinary regression. The differences are within a standard error; the lasso is chosen for using only 10 predictors.

The final test

final_lasso <- glmnet_wf |>
  finalize_workflow(best_penalty) |>
  last_fit(gpa_split, metrics = regression_metrics)

Off by 0.16 GPA points on average, capturing 65% of the variation.

Predicted against actual

Predictions are less spread than reality: extreme students are pulled towards the average.

How much does year one add to past grades?

gpa_only_wf <- workflow(recipe(final_gpa ~ gpa_s1 + gpa_s2, data = gpa_train),
                        linear_reg())

Past grades are by far the best predictor. Sleep, stress, and support matter (Chapter 8), but their influence is already in the grades.

In your field: engineering

lm_concrete  <- workflow(compressive_strength ~ ., linear_reg())
xgb_concrete <- workflow(compressive_strength ~ .,
  boost_tree(trees = 500, learn_rate = 0.05, tree_depth = 4) |>
    set_engine("xgboost") |> set_mode("regression"))

Concrete strength from its recipe and age: curves and interactions.

Boosting cuts RMSE from 10.6 to 4.5 MPa. Flexible models win when the data has structure for them.

Practical lab: the Chapter 13 playground

Work through the playground exercises in your browser, with hints and solutions.

The browser version builds ridge regression by formula; the download uses glmnet and XGBoost.

Practical exercises 1–3: errors and shrinkage

1. MAE and RMSE by hand; change one prediction by 1.0 and compare.
2. Repeat the 60-student experiment with 150 students.
3. Ridge against lasso coefficients: how many are exactly zero?

Practical exercises 4–6: design and communication

4. Scale scores instead of 22 items: does the lasso’s RMSE change?
5. XGBoost with 100, 500, and 1000 trees at learning rate 0.03.
6. Explain an RMSE of 0.2 to academic advisers in two sentences.

Try this yourself

Take a prediction problem from your own field.

• count your cases and your predictors;
• name a baseline prediction;
• decide whether a lasso, a ridge, or boosting suits the structure;
• plan how you would report the final test.

Troubleshooting guide (Part 1)

Troubleshooting guide (Part 2)

Completion checklist

• predictors are known at prediction time;
• a baseline is reported;
• RMSE, MAE, and R² are reported;
• predictors are normalised before regularising;
• penalty and mixture are tuned by cross-validation;
• selected predictors are not described as causes;
• the test set is used once.

Misconceptions to leave behind (Part 1)

Misconceptions to leave behind (Part 2)

The chapter in one sentence

With many predictors, shrink the coefficients or build from small trees, compare against a baseline by cross-validation, and never read a selected predictor as a cause.

Next: Chapter 14

The next chapter revisits clustering:

• what counts as a group;
• Gaussian mixture models and soft membership;
• clusters as dense regions: DBSCAN;
• evaluating a clustering;
• reporting the choices behind the clusters.

Questions

How many predictors, and how many cases, would your model have?

What is the simplest prediction your model must beat?