Playground: Chapter 13

Predictive Regression

This page practises the ideas of Chapter 13 in four parts. Practise the chapter works through the book’s exercises, with starter code, hints, and solutions. Go further goes beyond the book, with new questions and built-in datasets. Check your understanding asks short questions whose answers open with a click. Do it yourself gives open tasks with an empty code space and no answers: you build and judge the predictions yourself, as you will in your own thesis.

In the exercises below, replace each ______ with your own code and press Run Code.

Practise the chapter

These are the exercises at the end of Chapter 13, with the same numbers.

Exercise 1: MAE and RMSE by hand

Calculate the MAE and RMSE of the chapter’s tiny example by hand. Then change one prediction so that it is off by 1.0 GPA point, and explain which measure changes more, and why.

Solution

print("Hand: MAE =", round(np.mean(np.abs(errors)), 3), "RMSE =", round(np.sqrt(np.mean(errors**2)), 3))
# ...
print("After outlier: MAE =", round(np.mean(np.abs(errors)), 3), "RMSE =", round(np.sqrt(np.mean(errors**2)), 3))

The MAE is 0.22 and the RMSE 0.25, as in the chapter. With the last student’s prediction moved to 3.5, one miss of 1.0 point, the MAE rises to 0.34, but the RMSE almost doubles to 0.48. Squaring makes a 1-point error count 100 times as much as a 0.1-point error, whereas MAE counts it 10 times as much. Report the RMSE when large misses are especially costly, and the MAE when typical error is what matters; reporting both shows whether a few big misses drive the result.

Exercise 2: Sixty students or 150

Repeat the chapter’s 60-student experiment with 150 students, and report the gap between ordinary regression and the lasso.

Solution

for n in [60, 150]:

With 60 students, ordinary regression fits its own sample closely (RMSE ~ 0.12) but predicts new students with an RMSE around 0.32, no better than baseline (0.33). With 150 students, the test RMSE improves to ~0.23: more students give the 38 coefficients more information and less room to chase noise. In the chapter’s pipeline, the lasso gives 0.22 with 60 students and 0.21 with 150, against 0.66 and 0.26 for ordinary regression. The gap shrinks dramatically as the sample grows: regularisation matters most when the sample is small compared with the number of predictors.

Exercise 3: Ridge and lasso coefficients

Fit a ridge regression with the best ridge penalty, compare its coefficients with the lasso’s, and count how many are exactly zero.

This exercise is also demonstrated in the download project and in Go Further below.

Solution

# Ridge and Lasso comparison
from sklearn.linear_model import Ridge, Lasso
# In the chapter pipeline:
ridge_zero = np.sum(best_ridge_coefs == 0)
lasso_zero = np.sum(best_lasso_coefs == 0)

The best ridge model keeps all coefficients non-zero, while the lasso sets most coefficients to exactly zero. Both give the two year-one GPAs the largest coefficients. Ridge spreads the rest of the credit thinly across every predictor; the lasso gives it to a few and drops the others. Ridge keeps everything and shrinks; the lasso shrinks and selects.

Exercise 4: Scale scores instead of items

Replace the 22 questionnaire items with the four scale scores, as in Chapter 11, and describe how the cross-validated RMSE of the lasso changes.

Solution

With the four scale scores, the lasso’s cross-validated RMSE is ~0.203; with the 22 items, it was ~0.204: no difference worth mentioning. The individual items add no predictive information that the scale scores lack, which is what a good scale should achieve: averaging keeps the underlying signal and removes item noise. For prediction, the smaller model with four scores is simpler to explain and as accurate.

Exercise 5: More trees for Gradient Boosting

Tune Gradient Boosting with n_estimators = [100, 500, 1000] and a learning rate of 0.03, and describe what happens to the RMSE as trees are added.

Solution

gb_trees_cv = GridSearchCV(
    Pipeline([('prep', gpa_recipe), ('model', GradientBoostingRegressor(learning_rate=0.03, max_depth=1, random_state=2026))]),
    param_grid={'model__n_estimators': [100, 500, 1000]},
    cv=gpa_folds, scoring='neg_root_mean_squared_error'
).fit(X_train, y_train)

The cross-validated RMSE is ~0.224 with 100 trees, 0.223 with 500, and 0.223 with 1,000: nearly all the improvement comes in the first 100 trees, and after that each extra tree adds almost nothing. None beats the lasso (~0.204). With a small learning rate, boosting improves steadily, but when the underlying signal is predominantly linear, tree ensembles have little advantage over penalised linear models.

Exercise 6: Explaining the RMSE to advisers

Write two sentences for the academic advisers explaining what an RMSE of 0.2 means for an individual student’s predicted GPA. Write your answer first, then open the model answer.

Model answer

“The model’s predictions of final GPA are typically about 0.2 grade points away from the GPA a student actually achieves, so a student predicted to finish on 3.0 will usually end somewhere between about 2.8 and 3.2, and occasionally further away. The prediction is a useful starting point for an advising conversation, not a final verdict: two students with the same prediction can finish half a grade point apart.” (If errors are roughly normal, about two-thirds of students finish within one RMSE of their prediction and about 95% within two, from 2.6 to 3.4 in the example.)

Go further

These exercises go beyond the book.

Exercise 7: Ridge regression by hand

Ridge regression has an analytical formula: the coefficients are \((X^\top X + \lambda I)^{-1} X^\top y\), where \(I\) is the identity matrix. With \(\lambda = 0\), it is ordinary least squares. Calculate the ridge coefficients of the two year-one GPAs (standardised) for a range of penalties, and observe the shrinkage.

Solution

b = np.linalg.solve(X_ridge.T @ X_ridge + lam * I, X_ridge.T @ y_ridge)

With no penalty (\(\lambda = 0\)), the coefficients are ~0.120 for semester 1 and 0.154 for semester 2; with a penalty of 100 they shrink to 0.111 and 0.130, with 1,000 to 0.055 and 0.059, and with 10,000 almost to zero. Two things happen at once: both coefficients shrink, and they become more equal. Because the two GPAs correlate strongly (r = 0.69), ridge shares credit between them. Adding \(\lambda I\) also ensures numeric stability when predictors are collinear.

Exercise 8: One sample of 60, or ten?

The chapter’s 60-student result came from one random sample. Repeat the experiment with 10 different samples of 60, fitting ordinary regression with all 38 predictors versus only the two year-one GPAs, and compare test error variability.

Solution

print(res_df.std().round(3))

With all 38 predictors, the test RMSE fluctuates between ~0.27 and 0.39 across samples (SD ~ 0.036), averaging 0.34 (worse than baseline). With only the two GPAs, it stays steady between 0.197 and 0.205 (SD ~ 0.003). The flexible model is not only worse on average but far more variable: its performance is highly sensitive to the sample draw. This directly illustrates the variance component of the bias-variance trade-off.

Exercise 9: Boosting by hand

Boosting starts with the average, fits a one-split tree (stump) to the residuals, adds a fraction of its predictions (the learning rate), and iterates. Run 100 rounds, tracking RMSE on training and test sets.

Solution

pred_test += learn_rate * tree.predict(X_test_b)

Both errors fall quickly in the opening rounds. Test error reaches its minimum (~0.205) early on, after which training error continues descending (~0.187) while test error levels off. The widening gap signals the onset of overfitting, demonstrating why n_estimators is a crucial hyperparameter tuned via cross-validation.

Exercise 10: The shape of a tree

The black cherry tree dataset records girth, height, and timber volume for 31 trees. Compare two models for predicting volume with leave-one-out cross-validation: a linear model and a log-log model.

Solution

p_log = np.exp(m_log.predict(np.log(test_c[['Girth', 'Height']]))[0])

The straight-line model achieves a leave-one-out RMSE of 4.3 cubic feet; the log-log model achieves 2.6, about 40% lower error. Examining the log-log regression coefficients shows that volume scales with \(\text{Girth}^{2.0}\) and \(\text{Height}^{1.1}\), mimicking the geometric formula for a cylinder (\(V \propto r^2 h\)). Thoughtful feature transformation often outperforms complex black-box algorithms on physical phenomena.

Check your understanding

Answer each question in your own words first, then click to see a model answer.

1. Why does ordinary regression overfit when there are many predictors and few cases?

Answer

Because each predictor adds a free coefficient. With many coefficients and few observations, the model has sufficient flexibility to memorise idiosyncratic sample noise rather than true population patterns. Correlated predictors aggravate this by causing coefficients to inflate in magnitude with opposite signs.

2. What is the difference between ridge regression and the lasso?

Answer

Both add a penalty against large coefficients. Ridge uses an \(L_2\) penalty (sum of squared coefficients), shrinking coefficients continuously and sharing credit across correlated predictors without setting any to zero. Lasso uses an \(L_1\) penalty (sum of absolute coefficients), which drives some coefficients to exactly zero, performing automated variable selection.

3. What does the penalty \(\lambda\) (or \(\alpha\) in scikit-learn) do, and how is its value chosen?

Answer

It controls the penalty weight on coefficient size. When \(\alpha = 0\), it is ordinary least squares; as \(\alpha \to \infty\), all coefficients collapse to zero, predicting the global target mean. Because \(\alpha\) is a hyperparameter, it must be selected through \(K\)-fold cross-validation on the training set.

4. How does boosting differ from a random forest?

Answer

A random forest fits many deep, independent trees in parallel on bootstrap resamples and averages their predictions to reduce variance. Boosting fits shallow trees (weak learners) sequentially, where each successive tree targets the residual errors of the existing ensemble, reducing bias.

5. Why are a model’s predictions pulled towards the average, so that extreme achievers are predicted too low and low achievers too high?

Answer

Because an observed outcome reflects predictable underlying signal plus unobserved random noise (luck). The model can only fit the systematic signal, so extreme actual values inevitably regress towards the conditional mean. Regularisation accentuates this shrinkage.

Do it yourself

These tasks have no starter code and no answers. Each code space below is empty and ready to run: write your own code, as you would for a thesis. The data (gpa_data, train, and test) and function rmse() are already loaded.

1. Predict semester 4 wellbeing, rather than GPA, from year-one information. Construct the dataset with semester 4 wellbeing as target, compare a baseline, a linear regression with a few predictors, and Gradient Boosting, reporting test RMSE.

2. Choose a small model from Task 1 and explain, in a short paragraph, why its predictors are not a causal list of the drivers of wellbeing.

3. Write two sentences for student services explaining what your wellbeing model’s RMSE means for an individual student in the units of the wellbeing scale.

Work on your own computer

Download the Chapter 13 project

The project contains the datasets and all exercises in Python (exercises.py and solutions.py).

Download chapter13.zip, unzip it, and open the folder in your favourite editor (VS Code, PyCharm, etc.).