Playground: Chapter 12

Classification Models

This page practises the ideas of Chapter 12 in four parts. Practise the chapter works through the book’s exercises, with starter code, hints, and solutions. Go further goes beyond the book, with new questions and the Palmer Penguins dataset. Check your understanding asks short questions whose answers open with a click. Do it yourself gives open tasks with an empty code space and no answers: you choose how a classifier is used, as you will in your own thesis.

In the exercises below, replace each ______ with your own code and press Run Code.

Practise the chapter

These are the exercises at the end of Chapter 12, with the same numbers. In the setup, X_train and y_train hold the training students, and cv_predictions holds each student’s cross-validated probability of considering dropout from logistic regression.

Exercise 1: A tree of depth 2

Fit and display the decision tree for the wellbeing data with max_depth = 2. Name the predictors it uses, and explain the tree in plain words, as you would to a counsellor.

NoteHint

Set max_depth=2 to limit the tree to at most two questions before reaching a leaf.

TipSolution 
tree2_pipe = Pipeline([
    ("prep", tree_recipe),
    ("model", DecisionTreeClassifier(max_depth=2, min_samples_leaf=10, random_state=2026))
])
tree2_pipe.fit(X_train, y_train)

feature_names = [f.split("__")[-1] for f in tree2_pipe.named_steps["prep"].get_feature_names_out()]
fig, ax = plt.subplots(figsize=(8, 4))
plot_tree(tree2_pipe.named_steps["model"], feature_names=feature_names, class_names=["No", "Yes"], filled=True, ax=ax)
plt.show()

The tree splits first on support (below ~2.55) and then on burnout. In plain words: “Students who feel reasonably supported by their supervisor rarely consider dropping out. Among students who feel poorly supported, those who are also severely burnt out are the primary high-risk group to watch; more than half of them consider dropping out.”

Exercise 2: Tuning the forest’s max_features

Tune the random forest’s max_features (mtry) over [2, 5, 10] with cross-validation, and compare the best value with the default.

This exercise runs a 500-tree random forest across multiple folds, so it is best explored in the download project.

forest_pipe = Pipeline([
    ("prep", tree_recipe),
    ("model", RandomForestClassifier(n_estimators=500, random_state=2026))
])

forest_grid = GridSearchCV(
    forest_pipe,
    param_grid={"model__max_features": [2, 5, 10]},
    cv=dropout_folds,
    scoring="roc_auc"
)
forest_grid.fit(X_train, y_train)

forest_tuning = pd.DataFrame({
    "max_features": [2, 5, 10],
    "mean_auc": forest_grid.cv_results_["mean_test_score"],
    "std_err": forest_grid.cv_results_["std_test_score"] / np.sqrt(10)
})
print(forest_tuning.round(3))

The cross-validated AUCs are around 0.80 across max_features = 2, 5, 10, with differences much smaller than the standard errors (0.02). The default (sqrt of the number of predictors, around 4) is robust and near-optimal.

Exercise 3: Very many neighbours

Extend the k-NN grid to n_neighbors = [41, 81, 161, 301], describe what happens to the AUC, and explain what happens to a k-NN model as \(k\) approaches the number of training students.

This multi-model tuning grid is pre-computed in the download project.

knn_pipe = Pipeline([
    ("prep", dropout_recipe),
    ("knn", KNeighborsClassifier())
])

knn_grid = GridSearchCV(
    knn_pipe,
    param_grid={"knn__n_neighbors": [41, 81, 161, 301]},
    cv=dropout_folds,
    scoring="roc_auc"
)
knn_grid.fit(X_train, y_train)

knn_tuning = pd.DataFrame({
    "neighbors": [41, 81, 161, 301],
    "mean_auc": knn_grid.cv_results_["mean_test_score"],
    "std_err": knn_grid.cv_results_["std_test_score"] / np.sqrt(10)
})
print(knn_tuning.round(3))

With more neighbours, each prediction averages over more students and becomes smoother, which suits this dataset because risk follows a smooth gradient. But if \(k\) reached the full training set size (\(N = 405\)), every single student would receive the exact same average probability (15%), and the AUC would collapse to 0.50 (random guessing).

Exercise 4: The F1 score by hand

Using the tiny example’s confusion matrix (a questionnaire flags 12 of 100 students; 8 of the 12 are among the 10 truly at risk), calculate the F1 score yourself from precision and recall: \(F_1 = 2 \times \text{precision} \times \text{recall} / (\text{precision} + \text{recall})\).

NoteHint

Precision divides the true positives (8) by all the students flagged (12). Recall divides true positives by all students truly at risk (10). F1’s denominator is precision_tiny + recall_tiny.

TipSolution 
precision_tiny = 8 / 12
recall_tiny    = 8 / 10
f1 = 2 * precision_tiny * recall_tiny / (precision_tiny + recall_tiny)
print(f"Precision: {precision_tiny:.3f}, Recall: {recall_tiny:.3f}, F1: {f1:.3f}")

Precision is 0.667 and recall is 0.800, giving an F1 score of 0.727. F1 is the harmonic mean, which always sits closer to the lower of the two measures.

Exercise 5: Contacting only 10% of students

Suppose the counselling service can only contact 10% of new students. Build the threshold table from the cross-validated predictions, recommend a threshold, and report the recall and precision it would give.

NoteHint

Inspect the flagged column in the output table. Look for the highest threshold that keeps flagged <= 0.10.

TipSolution

At threshold 0.50, the model flags 8.4% of students, finding 28.4% of at-risk students with 50.0% precision. At threshold 0.40, it flags 12.4%, exceeding the 10% capacity limit. Therefore, threshold 0.50 (or a value near 0.45) is the highest threshold that stays within counselling capacity.

Exercise 6: Class weighting vs unweighted baseline

Compare class_weight='balanced' with the unweighted baseline, and compare sensitivity and AUC.

balanced_pipe = Pipeline([
    ("prep", dropout_recipe),
    ("model", LogisticRegression(class_weight="balanced", random_state=2026))
])

bal_res = cross_validate(
    balanced_pipe, X_train, y_train, cv=dropout_folds,
    scoring=["roc_auc", "recall", "precision"]
)
print("Balanced weighting - AUC:", round(bal_res["test_roc_auc"].mean(), 3),
      "Recall:", round(bal_res["test_recall"].mean(), 3),
      "Precision:", round(bal_res["test_precision"].mean(), 3))

At the default 0.5 threshold, balanced class weighting boosts sensitivity from 28.4% to 71.0%, but precision falls to 32.4%, and the AUC remains 0.81. Weighting acts like shifting the threshold downwards; it does not change the model’s fundamental ranking ability.

Exercise 7: A cheaper missed student

Repeat the cost analysis with a cost of 3 for a missed student instead of 10. Report the new cheapest threshold, and compare it with the decision-theory rule \(1 / (1 + 3) = 0.25\).

NoteHint

Set cost_miss = 3. The decision theory rule calculates cost_false_alarm / (cost_false_alarm + cost_miss).

TipSolution 
cost_miss        = 3
cost_false_alarm = 1

cost_rows = []
for t in np.arange(0.02, 0.62, 0.02):
    misses = np.sum((cv_predictions < t) & (y_train == 1))
    false_alarms = np.sum((cv_predictions >= t) & (y_train == 0))
    total_cost = cost_miss * misses + cost_false_alarm * false_alarms
    cost_rows.append({
        "threshold": round(t, 2),
        "total_cost": total_cost,
        "misses": misses,
        "false_alarms": false_alarms
    })

cost_df = pd.DataFrame(cost_rows)
best_cost_row = cost_df.loc[cost_df["total_cost"].idxmin()]
print("Cheapest threshold:", best_cost_row["threshold"])
print("Decision theory rule:", cost_false_alarm / (cost_false_alarm + cost_miss))

When missing a student costs only 3, the optimal threshold rises from ~0.04-0.10 up to ~0.26-0.30, in close agreement with the decision-theory rule (\(1 / (1 + 3) = 0.25\)).

Go further

Exercise 8: Confusion matrices by hand

Ten students have the predicted probabilities and true outcomes below. Count the four cells of the confusion matrix, and the recall and precision, at thresholds of 0.5, 0.35, and 0.12.

NoteHint

A student is flagged when their probability is at or above the threshold: prob >= t.

TipSolution 
flag = prob >= t

At 0.50: flags 4 students, 3 truly at risk (recall = 0.60, precision = 0.75). At 0.35: flags 6 students, 4 truly at risk (recall = 0.80, precision = 0.67). At 0.12: flags 9 students, finding all 5 at risk (recall = 1.00, precision = 0.56).

Exercise 9: When a false alarm has a cost

Suppose a flagged student receives a formal letter, which causes distress, so a false alarm costs 2 instead of 1, while a missed student still costs 10. Find the cheapest threshold.

NoteHint

Multiply false alarms by cost_false_alarm.

TipSolution 
fa_costs.append({"threshold": round(t, 2), "total_cost": cost_miss * m + cost_false_alarm * fa})

Doubling the cost of a false alarm raises the optimal threshold from ~0.09 to ~0.16-0.18, matching the decision-theory rule (\(2 / (2 + 10) = 0.167\)).

Exercise 10: Faculty subgroup performance

Check model recall and precision across faculties at threshold 0.20:

NoteHint

The code is complete: click Run Code to inspect how the classifier performs across different faculties.

TipSolution

Recall varies across faculties from ~50% to over 80%. Because each faculty has only 10 to 25 at-risk students in the training split, subgroup counts have wide sampling variability, but evaluating fairness across subgroups is essential before real-world deployment.

Exercise 11: Classifying Palmer Penguins species

Fit a random forest to the Palmer Penguins dataset and inspect its multiclass confusion matrix:

NoteHint

Set n_estimators=500 to train an ensemble of 500 decision trees.

TipSolution 
p_rf = RandomForestClassifier(n_estimators=500, random_state=1)

The random forest separates Adelie, Chinstrap, and Gentoo with near 98% test accuracy, showing how tree ensembles handle non-linear multidimensional morphological boundaries.

Check your understanding

1. What is the difference between precision and recall?

Recall (sensitivity) is the share of truly positive cases flagged: how many students at risk are successfully identified. Precision is the share of flagged cases that are truly positive: how often an intervention is justified.

2. Why is choosing a classification threshold a value judgement and not only a statistical decision?

Because the threshold determines how many errors of each type occur, and what those errors cost is a moral and practical judgement about human consequences: a missed student who drops out versus an unnecessary meeting.

3. What does a random forest average, and why does averaging help?

It averages the probability votes of hundreds of de-correlated decision trees, each trained on a bootstrap sample with random subsets of features. Because their individual errors are largely independent, averaging cancels out variance without adding bias.

4. Why does k-nearest neighbours need the predictors to be normalised?

Because k-NN calculates Euclidean distance. Without normalisation, features measured in large units (e.g. caffeine in milligrams) would dominate the distance, rendering 1-to-5 Likert scores completely irrelevant.

5. What does class weighting or upsampling change, and what does it not change?

It shifts the predicted probabilities upwards, raising sensitivity at the standard 0.50 threshold. It does not improve the model’s fundamental ranking ability (AUC stays almost identical), and it disrupts the calibration of the predicted probabilities.

Do it yourself

1. Invent a scenario for the dropout model: decide the costs of a miss and of a false alarm, calculate the cost-minimising threshold, and report recall and precision.

2. Compare three classifiers on a different binary outcome (such as lives_away) with cross-validation, and argue for one based on AUC and interpretability.

3. Write the paragraph that reports a classifier for a thesis: model, cross-validation protocol, AUC, chosen threshold and rationale, and recall and precision at that threshold.

Work on your own computer

NoteDownload the Chapter 12 project

The project contains the datasets and exercises as runnable Python scripts.

  • Download chapter12.zip, unzip it, and open the folder in your favourite Python editor (VS Code, PyCharm, or Jupyter).