Playground: Chapter 11
Introduction to Machine Learning in Python
This page practises the ideas of Chapter 11 in four parts. Practise the chapter works through the book’s exercises with scikit-learn pipelines, starter code, hints, and solutions. Go further explores data leakage, manual cross-validation loops, noise simulations, and passenger survival on the Titanic. Check your understanding asks short questions whose answers open with a click. Do it yourself gives open tasks with an empty code space and no answers: you design the test of a model yourself, as you will in your own thesis.
In the exercises below, replace each ______ with your own code and press Run Code.
Practise the chapter
These are the exercises at the end of Chapter 11, with the same numbers.
Exercise 1: The model without the questionnaire
Remove the questionnaire scores (stress, burnout, support, satisfaction) from the training data and fit the logistic regression pipeline again with 10-fold cross-validation. Report how much the cross-validated AUC falls, and what that says about the questionnaire.
The metric to score across validation folds is "roc_auc".
no_q_auc = cross_val_score(pipe_no_q, X_train_no_q, y_train, cv=cv, scoring="roc_auc").mean()
print(f"Full CV AUC: {full_auc:.3f}, Without Questionnaire: {no_q_auc:.3f}")With every predictor, the 10-fold cross-validated AUC is ~0.806; without the four questionnaire scores, it drops to ~0.714. The background demographics and first-semester performance still predict dropout moderately well, but the psychological questionnaire carries a large share of the signal: students’ subjective feelings of stress, burnout, and support tell more about risk than their faculty, age, or grades alone.
Exercise 2: A different split
Split the data 80/20 instead of 75/25 with seed 7, and describe how much the test AUC of the logistic model changes, and why it might.
An 80/20 split uses test_size=0.20.
X_tr80, X_te80, y_tr80, y_te80 = train_test_split(
X, y, test_size=0.20, stratify=y, random_state=7
)The 75/25 split gives a test AUC of ~0.843, while the 80/20 split gives ~0.847. In a sample of 600 students, a 20% test partition contains only 120 students, of whom only about 18 consider dropping out. With so few positive cases, small differences in which individual cases end up in the test set produce natural fluctuation. This is why cross-validation—which averages over multiple folds—is preferred for model selection.
Exercise 3: Tuning k-nearest neighbours
Tune k-nearest neighbours over n_neighbors = [5, 11, 21, 41] with 10-fold cross-validation. Identify the best value, and compare its AUC with logistic regression.
Specify scoring="roc_auc" in GridSearchCV.
knn_grid = GridSearchCV(knn_pipe, param_grid=param_grid, cv=cv, scoring="roc_auc")
knn_grid.fit(X_train, y_train)The cross-validated AUC rises steadily as \(k\) increases, reaching its peak around \(k = 41\) (~0.796). Larger \(k\) smooths out local sample noise. However, even the best \(k\)-NN model remains slightly below logistic regression (~0.806), because the true risk relationship in this dataset is smooth and roughly monotonic, which linear models capture parsimoniously.
Exercise 4: Zero-variance predictors
Check if any columns in the preprocessed feature matrix have zero variance using VarianceThreshold(). When is this check useful?
VarianceThreshold(threshold=0.0) removes features that have the same constant value across all rows.
X_prep = preprocessor.fit_transform(X_train)
zv_selector = VarianceThreshold(threshold=0.0)
X_zv = zv_selector.fit_transform(X_prep)Both before and after, there are 24 features: no predictor in the training set is constant. Zero-variance predictors carry no information and can cause numerical division-by-zero errors in scalers or algorithms. A variance threshold is a useful automated guardrail whenever data subsets, filters, or rare categorical levels might collapse into constant columns.
Exercise 5: Explaining accuracy to an administrator
In two or three sentences, explain to a university administrator why a model with 85% accuracy might still be useless for finding students at risk. Write your answer first, then open the model answer.
“Only about 15% of our students consider dropping out, so a naive ‘model’ that simply predicts that nobody will is already right 85% of the time, while detecting zero students at risk. Accuracy mostly rewards getting the large group of safe students right. What matters for early intervention is how many truly at-risk students the model identifies (sensitivity) and how well it ranks risk (ROC AUC), not overall accuracy.”
Exercise 6: More training points
Repeat the polynomial simulation with 50 training points instead of 15. Describe how the error curve on new data changes, and explain why more data allows a more flexible model.
Generate 50 points by setting n = 50 in make_points(50, seed=3).
train_50 = make_points(50, seed=3)With 15 points, test RMSE exploded for high polynomial degrees (e.g. degree 12). With 50 training points, the test error curve remains stable and low across degrees 3 to 12 (~0.295 to 0.305). With more data, a flexible curve is constrained by many observed points, preventing it from bending wildly through individual noise artifacts. More data allows models of higher complexity to generalize well.
Go further
These exercises go beyond the book.
Exercise 7: Data leakage
A researcher adds 200 columns of pure random noise, picks the 5 that correlate most strongly with dropout, and evaluates a model. Compare feature selection performed on all data versus on the training data only.
Feature correlations must be evaluated on the 200 noise features using only tr_idx. Replace ______ with 200.
corrs_tr = np.array([np.corrcoef(noise[tr_idx, j], y.iloc[tr_idx])[0, 1] for j in range(200)])Selecting features on the whole dataset produces an artificially inflated test AUC (~0.64) from pure Gaussian noise! By peeking at the test set during feature selection, noise features that happened by chance to align with test labels were selected. When feature selection is strictly restricted to training data, the test AUC is ~0.50, correctly reflecting zero predictive signal.
Exercise 8: Cross-validation by hand
Implement 5-fold cross-validation manually with a loop over StratifiedKFold.
In each iteration, train_idx contains the 4 training folds and val_idx contains the 1 validation fold.
print("Fold AUCs:", np.round(fold_scores, 3))
print(f"Mean manual CV AUC: {np.mean(fold_scores):.3f}")The five folds yield AUCs between ~0.75 and 0.87, averaging ~0.801. Every training observation serves as a held-out evaluation case exactly once.
Exercise 9: Noisier data, simpler model
Repeat the polynomial simulation with 15 training points, but double the noise standard deviation to sd = 0.6. At which degree is the test RMSE now lowest?
Set sd=0.6 in both calls to make_noisy.
train_noisy = make_noisy(15, sd=0.6)
test_noisy = make_noisy(300, sd=0.6)With twice as much noise, the optimal polynomial degree drops to degree 1 or 2 (a simple line or gentle slope). When noise overwhelms the signal, flexible polynomials fit noise aggressively, making simple models far more robust out of sample.
Exercise 10: Who survived the Titanic?
Fit a logistic regression model predicting passenger survival on the Titanic from Class, Sex, and Age. Compare its test AUC and accuracy against a baseline predicting that nobody survived.
OneHotEncoder encodes passenger demographic categories into dummy predictors.
print(f"Titanic Test AUC: {roc_auc_score(y_te_t, prob_titanic):.3f}")
print(f"Titanic Test Accuracy: {accuracy_score(y_te_t, pred_titanic):.3f}")
print(f"Majority Class Base: {(y_te_t == 0).mean():.3f}")The model achieves a test ROC AUC of ~0.746 and an accuracy of 77.1%, outperforming the 67.7% baseline of predicting no survivors.
Check your understanding
Answer each question in your own words first, then click to see a model answer.
1. What is the difference between a model built for prediction and one built for explanation?
A model for explanation asks why: which variables relate to the outcome, how strongly, and whether the relationship could be causal; its parameter estimates and confidence intervals matter. A model for prediction asks how well: how accurately it forecasts new, unseen cases. A good predictor can be useless for explanation, and a good explanation can predict poorly.
2. Why is the test set used only once?
Because every time a decision is made based on test set results (such as tweaking hyperparameters or feature subsets), information from the test set leaks into model development. The test set stops being truly unseen data, leading to overoptimistic performance claims.
3. What does cross-validation estimate?
Cross-validation estimates how well the entire modeling procedure (preprocessing, feature selection, and algorithm) will generalize to new data, computed entirely from the training partition.
4. What is overfitting, in one sentence?
Overfitting occurs when a model learns the random sample-specific noise alongside the real pattern, performing excellently on training data but poorly on new data.
5. What is data leakage?
Data leakage occurs when information outside the training partition—such as test set distributions, future information, or target values—is accidentally incorporated during preprocessing or feature selection.
Do it yourself
These tasks have no starter code and no answers. Each code space below is empty and ready to run: write your own code, as you would for a thesis. The datasets (students, semesters, questionnaire, dropout_data, titanic) and standard libraries (pandas, numpy, sklearn) are already imported.
1. Build a predictive model for another binary outcome, such as lives_away: prepare the data with a ColumnTransformer, split it 75/25, compare two algorithms with 10-fold cross-validation, and report the chosen model’s test ROC AUC alongside the majority-class accuracy.
2. Using the numbers from your model in Task 1, explain in a short paragraph why accuracy would or would not be misleading for your outcome.
3. Write the prediction section of a methods chapter for your model in Task 1: describe the predictors, the stratified split, cross-validation protocol, and the evaluation metric.
Work on your own computer
The project contains the data and all four parts of this page as a clean Python project: the exercises with blanks, questions to check understanding (answers in solutions.py), and the open tasks.
- Download chapter11.zip, unzip it, and open the folder in VS Code or your favourite Python editor.
- Install the required packages:
pip install pandas numpy scikit-learn matplotlib seaborn