By the end of today you can

• explain how trees, random forests, k-nearest neighbours, and SVMs classify;
• fit each with scikit-learn, and compare them fairly;
• read a confusion matrix;
• calculate sensitivity, specificity, precision, and F1;
• choose a threshold from the costs of the two errors;
• handle an imbalanced outcome, and say what resampling changes.

Two questions about a classifier

1. Would more flexible methods rank the cases better than logistic regression?
2. What happens when predictions become decisions?

Every decision rule flags some cases wrongly and misses others. The balance depends on what each error costs, and to whom.

Same data, same split, same folds

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=2026
)

dropout_recipe = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")), ("scaler", StandardScaler())]), num_cols),
    ("cat", OneHotEncoder(drop="first", sparse_output=False, handle_unknown="ignore"), cat_cols)
])

dropout_folds = StratifiedKFold(n_splits=10, shuffle=True, random_state=2026)

Exactly as in Chapter 11, so every model is trained and tested on the same students.

A decision tree asks questions

tiny_tree = DecisionTreeClassifier(min_samples_split=2, criterion="gini", random_state=2026)
tiny_tree.fit(tiny[["stress", "support"]], (tiny["dropout"] == "Yes").astype(int))

Twelve imaginary students, four considering dropout:

1. “Is support \(\le\) 2.75?” puts seven, all “No”, on one side.
2. “Is stress > 3.1?” separates the rest perfectly.

How a tree chooses its questions

The tree tries every predictor and cut-off, and picks the one that makes the child groups most pure.

Why trees are attractive

• easy to explain;
• no dummy variables or normalisation needed;
• interactions captured automatically: stress matters only for low-support students.

tree_recipe = ColumnTransformer([
    ("num", SimpleImputer(strategy="median"), num_cols),
    ("cat", OneHotEncoder(drop="first", sparse_output=False, handle_unknown="ignore"), cat_cols)
])

A depth-3 tree for dropout

The top split is on support. Students meeting each condition branch to the left.

A single tree is unstable

A slightly different sample of students can produce a completely different tree.

A deep tree overfits (Chapter 11).

The next method turns this weakness into a strength.

Random forests

• grow hundreds of trees;
• each on a bootstrap sample of the students;
• each split chooses from a random subset of predictors (max_features);
• let the trees vote: \(P(\text{Yes})\) is the share voting “Yes”.

Each tree overfits in its own way; many errors cancel out. An ensemble of trees (Breiman 2001).

Fitting a forest

forest_pipe = Pipeline([
    ("prep", tree_recipe),
    ("model", RandomForestClassifier(n_estimators=500, random_state=2026))
])
forest_fit = forest_pipe.fit(X_train, y_train)

500 trees cannot be drawn: a black box. Permutation importance opens it a little.

What the forest relies on

Shuffle one predictor, measure the drop in accuracy. Importance says what the model uses, not what causes dropout.

k-nearest neighbours

tiny["distance"] = np.sqrt((tiny["stress"] - 3.5)**2 + (tiny["support"] - 3.0)**2)
tiny.sort_values("distance").head(3)

A new student with stress 3.5 and support 3.0.

The three nearest students vote: two considered dropping out, so \(P(\text{Yes}) = 2/3\).

The three nearest neighbours

Distances need predictors on one scale: the pipeline normalises them with StandardScaler.

Tuning k

knn_grid = GridSearchCV(
    Pipeline([("prep", dropout_recipe), ("knn", KNeighborsClassifier())]),
    param_grid={"knn__n_neighbors": [1, 5, 11, 21, 41, 81]},
    cv=dropout_folds, scoring="roc_auc"
)

The smoother k-NN becomes, the better: the pattern in this data is smooth.

Support vector machines

• separate the classes with the widest empty “street”, the margin;
• only the students closest to the boundary, the support vectors, place it;
• overlapping classes: some students may cross, at a cost (\(C\));
• a kernel lets the boundary bend; the RBF kernel is popular.

svm_linear = SVC(kernel="linear", probability=True, random_state=2026)
svm_rbf    = SVC(kernel="rbf", probability=True, random_state=2026)

Four ways to draw a boundary

Smooth gradient, rectangular blocks, patches, rounded regions. All agree: highest risk at high stress and low support.

Comparing six models fairly

candidate_models = {
    "logistic": Pipeline([("prep", dropout_recipe), ("model", LogisticRegression())]),
    "svm_linear": Pipeline([("prep", dropout_recipe), ("model", SVC(kernel="linear", probability=True))]),
    "forest": Pipeline([("prep", tree_recipe), ("model", RandomForestClassifier(n_estimators=500))]),
    "svm_rbf": Pipeline([("prep", dropout_recipe), ("model", SVC(kernel="rbf", probability=True))]),
    "knn": Pipeline([("prep", dropout_recipe), ("model", KNeighborsClassifier(n_neighbors=41))]),
    "tree": Pipeline([("prep", tree_recipe), ("model", DecisionTreeClassifier(max_depth=4))]),
}

# Cross-validate each candidate pipeline on the identical 10 folds

Same folds for every model. Trees use the simple recipe, the others the normalised one.

The simple model wins again

Risk changes smoothly, with no thresholds for a tree to find. Logistic regression is kept, and explained in one sentence.

From ranking to deciding

The AUC measures how well a model ranks students.

The counselling service needs a decision: contact this student or not.

A student is flagged when \(P(\text{Yes})\) is above a threshold, 0.5 by default. The confusion matrix counts how the decisions turn out.

The four outcomes

A screening questionnaire given to 100 students, 10 truly at risk.

Measures from the four counts

The study’s model at 0.5

cm = confusion_matrix(y_test, y_pred_te, labels=[1, 0])
# [[8, 15], [3, 124]]

Of 23 at-risk test students, only np.int64(8) are flagged. The model ranks well; the default 0.5 threshold is the problem.

0.5 is a default, not a law

Here a flag means a conversation: cheap and harmless. Missing a student who leaves is costly. The threshold belongs well below 0.5.

Choose the threshold without the test set

cv_predictions = cross_val_predict(
    logistic_pipe, X_train, y_train, cv=dropout_folds, method="predict_proba"
)[:, 1]

Cross-validated predictions for every training student, each from a model that did not see them.

The ROC curve

Every threshold at once. The AUC is the area under the curve.

The trade-off in a table

|   threshold | flagged   | recall   | precision   | specificity   |
|------------:|:----------|:---------|:------------|:--------------|
|        0.5  | 8%        | 28%      | 50%         | 95%           |
|        0.4  | 12%       | 36%      | 43%         | 92%           |
|        0.3  | 19%       | 52%      | 40%         | 86%           |
|        0.2  | 26%       | 63%      | 36%         | 80%           |
|        0.15 | 32%       | 69%      | 32%         | 74%           |
|        0.1  | 42%       | 82%      | 29%         | 65%           |

A lower threshold flags more students, finding more at risk at the cost of lower precision.

Costs of the two errors

cost_miss        = 10   # a student at risk who is not contacted
cost_false_alarm = 1    # an unnecessary conversation

Costs are judgements about consequences, not statistics. Make them explicit.

The cheapest threshold

Lowest at about np.float64(0.04), flagging 60% of students. Decision theory: \(1 / (1 + 10) \approx 0.09\). Both say: far below 0.5.

The threshold is a decision with an ethical side

• set costs with the people who will use the model;
• state them in the thesis;
• examine the effects on different groups;
• respect practical limits: counsellors cannot see everyone.

The counselling service settles on 0.2: flags about 26%, finds about 63% of those at risk, precision about 36%.

Applying 0.2 to the test students

flags_te = (y_prob_te >= 0.2).astype(int)
cm_02 = confusion_matrix(y_test, flags_te, labels=[1, 0])
# [[17, 6], [29, 98]]

The model now finds np.int64(17) of 23 at-risk students instead of np.int64(8), flagging 46 in all.

Accuracy fell to 77%. It does not matter: accuracy was never the goal.

Report the threshold

Sensitivity, specificity, precision, and F1 all depend on the threshold.

Report which threshold was used, and why.

The AUC does not depend on it, which is why it is used to compare models.

Imbalanced outcomes

Scikit-learn models support class weights directly via class_weight='balanced'.

Class weighting / Upsampling

balanced_pipe = Pipeline([
    ("prep", dropout_recipe),
    ("model", LogisticRegression(class_weight="balanced", random_state=2026))
])

Weighting applies to training loss. It pushed probabilities up, like lowering the threshold, but did not improve ranking (AUC 0.80).

After weighting, an output score of 0.6 no longer corresponds to a 60% empirical rate.

Does the model work equally well for everyone?

Similar shares, but too few students in the test set to detect subtle differences.

Check sensitivity and precision for every demographic group that matters, on as much data as possible.

In your field: ecology

penguins = dt.load_penguins().dropna(subset=["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g"])
p_rf = RandomForestClassifier(n_estimators=500, random_state=1)
p_rf.fit(p_X_train, p_y_train)
pd.crosstab(p_y_test, p_rf.predict(p_X_test))

Three penguin species from four morphological measurements: near 98% test accuracy.

Well-separated non-linear classes suit flexible tree ensembles.

Practical lab: the Chapter 12 playground

Work through the playground exercises in your browser, with hints and solutions.

The browser version runs Pyodide in WebAssembly; the download project runs in your local editor.

Practical exercises 1–4: models

1. A depth-2 tree, explained to a counsellor.
2. Tune the forest’s max_features over 2, 5, and 10.
3. k-NN with 81, 161, and 301 neighbours: what happens?
4. The F1 score of the tiny confusion matrix, by hand.

Practical exercises 5–7: decisions

5. Only 10% of students can be contacted: which threshold?
6. Class weighting instead of unweighted: sensitivity and AUC?
7. A missed student costs 3, not 10: the new cheapest threshold?

Try this yourself

For a classifier in your own field:

• name what a false positive and a false negative mean in practice;
• estimate the cost of each;
• calculate the decision-theory threshold;
• say who should agree to those costs before the model is used.

Troubleshooting guide (Part 1)

Troubleshooting guide (Part 2)

Completion checklist

• every model uses the same folds;
• preprocessing pipelines match each model’s needs;
• models are compared by cross-validated AUC;
• the confusion matrix is read, not just accuracy;
• costs of both errors are stated;
• the threshold is chosen from cross-validated predictions;
• the threshold is reported;
• performance is checked for each subgroup.

Misconceptions to leave behind (Part 1)

Misconceptions to leave behind (Part 2)

The chapter in one sentence

Compare models by how well they rank cases, then choose the threshold from what each kind of error costs, and report it.

Next: Chapter 13

The next chapter predicts a number:

• measuring prediction error;
• ordinary regression with many predictors;
• regularised regression: ridge, lasso, elastic net;
• boosting;
• comparing models and the final test.

Questions

In your field, which is worse: a false alarm or a missed case?

Who decides?