11 Introduction to Machine Learning in Python

Some research questions are not about why something happens, but about what will happen. A hospital wants to know which patients are likely to be readmitted, a bank which loans are likely to fail, a university which students are likely to struggle. The goal is not to understand the causes, although that may help, but to make accurate predictions for new cases, early enough to act on them. Such questions need a different standard of evidence. For explanation, a model is judged by whether its coefficients are well estimated and make sense. For prediction, a model is judged by one thing only: how well it works on cases it has never seen.

Machine learning is the set of methods and practices for building predictive models and testing them fairly. This chapter introduces its core ideas: the difference between prediction and explanation, why a model must be tested on new data, how models overfit, and how cross-validation and tuning choose a model honestly. It uses Python’s industry-standard scikit-learn framework, whose components are best understood as the design of a fair test. In the story, Chapter 8 used logistic regression to explain who considers dropping out; Elaf’s supervisor now asks whether the university could identify, at the end of a student’s first semester, the students at risk, so that help could be offered in time (RQ9).

TipBy the end of this chapter you will be able to

• Explain supervised and unsupervised learning, and prediction versus explanation as different kinds of research question.
• Explain generalisation, overfitting, and the bias-variance trade-off, and show them by simulation.
• Split data into training and test sets, and explain why a model must be tested on data it has not seen.
• Prepare data for modelling with ColumnTransformer pipelines, and combine preprocessing and models in a unified pipeline.
• Evaluate a classification model, and explain why accuracy alone can mislead.
• Use cross-validation to estimate performance honestly, tune a model’s hyperparameters with GridSearchCV, and evaluate the final model on the test set.

Teaching & Practice: Interactive Lecture Slides · Browser Playground Exercises

11.1 Machine learning and prediction

Machine learning covers methods that learn patterns from data in order to make predictions or find structure. In supervised learning, the data includes the outcome to be predicted, and the model learns the relationship between the predictors and that outcome. If the outcome is a category, such as “considering dropout: yes or no”, it is a classification problem; if it is a number, such as next semester’s GPA, it is a regression problem. Chapters 11 to 13 and Chapter 15 are about supervised learning. In unsupervised learning, there is no outcome, and the model looks for structure in the data itself. The clustering of Chapter 9 is unsupervised, and Chapter 14 returns to it.

Many machine learning methods are familiar statistics: logistic regression is both a statistical model and one of the most useful classifiers. What changes is the goal, and with it the way models are judged. Table 11.1 sets the two aims side by side.

Table 11.1: Explanation and prediction

	Explanation (Chapters 7 to 10)	Prediction (Chapters 11 to 16)
Question	Why does something happen?	What will happen for a new case?
Judged by	Sensible, well-estimated effects	Accuracy on new data
Typical models	Simple, interpretable	Anything that predicts well
Main danger	Confounding	Overfitting

A predictive model does not need to be causally correct to be useful: a model can predict dropout well from variables that do not cause it. Equally, a model that explains well may predict poorly, because the effects it estimates are small compared with everything it cannot see. Chapter 5 described prediction as a third aim of analysis, alongside description and explanation; this part of the book develops it.

11.2 Generalisation and overfitting

The central idea of machine learning is generalisation: a model is useful only if what it learned from one set of data carries over to new data. Any dataset contains two things, the pattern that would appear again in new data and the noise that belongs to this sample only. A model that learns the pattern generalises; a model that also learns the noise overfits, and looks better on its own data than it will ever be on new data.

A simulation makes the idea visible. The code below draws 15 points from a smooth curve with random noise added, and fits three models of increasing flexibility: a straight line, a gentle curve, and a very flexible curve (polynomials of degree 1, 3, and 12). It then draws new points from the same process, within the same range, to see how each model does on data it has not seen:

true_curve = lambda x: np.sin(2 * x)

def make_points(n, seed=None):
    if seed is not None:
        np.random.seed(seed)
    x = np.random.uniform(0, 3, n)
    y = true_curve(x) + np.random.normal(0, 0.3, n)
    return pd.DataFrame({"x": x, "y": y})

train_points = make_points(15, seed=3)
new_points = make_points(1000, seed=4)
new_points = (
    new_points[
        (new_points["x"] >= train_points["x"].min()) &
        (new_points["x"] <= train_points["x"].max())
    ]
    .iloc[:300]
    .reset_index(drop=True)
)

x_grid = np.linspace(train_points["x"].min(), train_points["x"].max(), 300)
curves_dict = {}

for d in [1, 3, 12]:
    poly_model = make_pipeline(PolynomialFeatures(d), LinearRegression())
    poly_model.fit(train_points[["x"]], train_points["y"])
    curves_dict[f"Degree {d}"] = poly_model.predict(x_grid[:, None])

fig, axes = plt.subplots(1, 3, figsize=(8, 3.2), sharey=True)

for ax, d in zip(axes, [1, 3, 12]):
    ax.scatter(new_points["x"], new_points["y"], color="grey", alpha=0.35, s=8, label="New points")
    ax.scatter(train_points["x"], train_points["y"], color="black", s=25, zorder=4, label="Training points")
    ax.plot(x_grid, curves_dict[f"Degree {d}"], color="#b2182b", linewidth=1.5, label=f"Degree {d}")
    ax.set_title(f"Degree {d}")
    ax.set_ylim(-2, 2)
    ax.set_xlabel("x")

axes[0].set_ylabel("y")
plt.tight_layout()
plt.show()

Figure 11.1: Three models fitted to the same 15 training points (black). The straight line is too simple, the degree-3 curve follows the pattern, and the degree-12 curve passes close to every training point but wanders far from the new points (grey).

The straight line is too simple to follow the wave: it underfits. The degree-3 curve captures the pattern. The degree-12 curve bends to pass close to every training point, and in doing so follows the noise, swinging far away from where new points actually lie. Measured on the training points, it would look like the best model; measured on new points, it is the worst. The prediction error of every degree from 1 to 12 shows the general pattern:

errors = []
for d in range(1, 13):
    model = make_pipeline(PolynomialFeatures(d), LinearRegression())
    model.fit(train_points[["x"]], train_points["y"])
    tr_err = root_mean_squared_error(train_points["y"], model.predict(train_points[["x"]]))
    te_err = root_mean_squared_error(new_points["y"], model.predict(new_points[["x"]]))
    errors.append({"degree": d, "training": tr_err, "new_data": te_err})

errors_df = pd.DataFrame(errors)
print(errors_df.head(6).round(3))

   degree  training  new_data
0       1     0.462     0.470
1       2     0.379     0.414
2       3     0.257     0.297
3       4     0.257     0.297
4       5     0.251     0.327
5       6     0.251     0.326

fig, ax = plt.subplots(figsize=(6, 3.6))
ax.plot(errors_df["degree"], errors_df["training"], marker="o", color="grey", linewidth=1.5, label="Training points")
ax.plot(errors_df["degree"], errors_df["new_data"], marker="o", color="#b2182b", linewidth=1.5, label="New points")
ax.set_yscale("log")
ax.set_xticks(range(1, 13))
ax.set_xlabel("Flexibility (polynomial degree)")
ax.set_ylabel("Prediction error (RMSE, log scale)")
ax.legend()
plt.tight_layout()
plt.show()

Figure 11.2: Prediction error (RMSE) on the training points and on new points, for polynomials of degree 1 to 12. Training error keeps falling as the model becomes more flexible; error on new data falls, then rises.

The training error falls steadily as the model becomes more flexible: a more flexible model can always fit its own data more closely. The error on new data falls at first, reaches its minimum at degree np.int64(3), and then rises. This U shape is known as the bias-variance trade-off. A model that is too simple has high bias: it misses part of the real pattern, whatever data it is given. A model that is too flexible has high variance: it changes a great deal from one sample to another, because it follows each sample’s noise. The best predictions come from a model in between, and the only way to find it is to measure performance on data the model did not learn from. The rest of the chapter builds that measurement into every step.

11.3 Designing a fair test

The scikit-learn framework provides a principled, modular design that makes a fair test of a predictive model clean and reliable. Each module handles one part of the pipeline: train_test_split partitions the data, ColumnTransformer and pipelines prepare it, estimators specify models, GridSearchCV tunes them, and metrics assesses performance on unseen data.

Figure 11.3 shows how the pieces fit together: the test data is set aside before anything else happens, all choices are made with the training data alone, and the test data is used once, at the end. The rest of the chapter works through each step.

flowchart TB
  A[All data] --> B[Split:<br/>train_test_split]
  B --> C[Training data]
  B --> T[Test data,<br/>set aside]
  C --> D[Preprocessor:<br/>imputation, encoding, scaling]
  D --> E[Model Pipeline:<br/>fit and tune with<br/>cross-validation]
  E --> F[Final model]
  T --> G[Evaluate once<br/>on the test data]
  F --> G

Figure 11.3: The machine learning workflow in scikit-learn.

11.4 Data for prediction

A predictive model can only use information that would be available when the prediction is made. The university wants to act at the end of the first semester, so the model may use the students’ background, their questionnaire scores from the start of the year, and their first-semester records, but nothing from later. The questionnaire scores are calculated as before:

scores = pd.DataFrame({
    "student_id": questionnaire["student_id"],
    "stress": questionnaire[["stress_1", "stress_2", "stress_3", "stress_5", "stress_6"]].assign(stress_4=6 - questionnaire["stress_4"]).mean(axis=1),
    "burnout": questionnaire.filter(like="burnout_").mean(axis=1),
    "support": questionnaire.filter(like="support_").mean(axis=1),
    "satisfaction": questionnaire.filter(like="satisfaction_").mean(axis=1),
})

sem1 = semesters[semesters["semester"] == 1].drop(columns=["semester"])

dropout_data = (
    students
    .merge(scores, on="student_id")
    .merge(sem1, on="student_id")
    .drop(columns=["student_id", "supervisor_id", "workshop", "workshop_sessions"])
)

print("Data shape:", dropout_data.shape)

Data shape: (600, 21)

Two details matter. The outcome is coded as binary (\(1\) for “Yes”, \(0\) for “No”), making “Yes” (considering dropout) the positive event to detect. And the workshop variables are removed: the workshop took place after the first semester, so it would not be known when the prediction is made. Using information that would not be available at the time of prediction is a form of data leakage, and it makes a model look better than it can ever be in practice.

counts = dropout_data["considering_dropout"].value_counts()
shares = dropout_data["considering_dropout"].value_counts(normalize=True)
pd.DataFrame({"count": counts, "share": shares.round(3)})

	count	share
considering_dropout
No	510	0.85
Yes	90	0.15

Only about 15% of students answer “Yes”. Such an imbalanced outcome is common in real prediction problems, from rare diseases to fraud, and, as the evaluation below shows, it catches out anyone who judges a model by accuracy alone.

11.5 Splitting the data

The most important rule of machine learning follows from the simulation above: never judge a model on the data it learned from. The first step is therefore to split the data. The training set is used to build and tune the model. The test set is locked away and used only once, at the very end, to estimate how the final model will perform on new students. The function train_test_split() with stratify=y makes sure that both sets have the same share of “Yes” answers:

X = dropout_data.drop(columns=["considering_dropout"])
y = (dropout_data["considering_dropout"] == "Yes").astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=2026
)

print(f"Training observations: {len(X_train)}")
print(f"Testing observations:  {len(X_test)}")

Training observations: 450
Testing observations:  150

Three quarters of the students are used for training and a quarter are set aside for the final test.

11.6 Preparing the data

Most models need the data prepared first: missing values filled in, categories turned into numbers, and numeric predictors put on a common scale. A ColumnTransformer bundles these steps cleanly:

num_cols = X.select_dtypes(include=[np.number]).columns.tolist()
cat_cols = X.select_dtypes(exclude=[np.number]).columns.tolist()

preprocessor = ColumnTransformer([
    ("num", Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler())
    ]), num_cols),
    ("cat", Pipeline([
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("ohe", OneHotEncoder(drop="first", sparse_output=False, handle_unknown="ignore"))
    ]), cat_cols)
])

The numeric pipeline replaces missing values with the variable’s median (SimpleImputer(strategy="median")) and standardizes predictors to z-scores (StandardScaler()). The categorical pipeline imputes missing categories with the most frequent value and creates one-hot dummy variables (OneHotEncoder(drop="first")), dropping the reference level just as linear regression did in Chapter 8.

To see what the preprocessor produces, fit_transform() estimates the parameters from the training set and transforms it:

X_train_prep = preprocessor.fit_transform(X_train)
feature_names = preprocessor.get_feature_names_out()
print("Transformed training shape:", X_train_prep.shape)
print("First 8 prepared feature names:", feature_names[:8].tolist())

Transformed training shape: (450, 24)
First 8 prepared feature names: ['num__age', 'num__financial_worry', 'num__stress', 'num__burnout', 'num__support', 'num__satisfaction', 'num__gpa', 'num__sleep_hours']

ImportantNo peeking at the test data

The medians, means, and standard deviations are estimated from the training data only, and then applied unchanged to the test data. If they were calculated from all the data, information from the test set would leak into the model, and the test would no longer be a fair one. Packaging preprocessing inside a Pipeline handles this automatically, which is one of the best reasons to use scikit-learn pipelines.

11.7 Specifying and fitting a model

A Pipeline combines the preprocessor and the classifier so they always travel together:

logistic_pipe = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000, random_state=2026))
])

logistic_pipe.fit(X_train, y_train)

Pipeline(steps=[('preprocessor',
                 ColumnTransformer(transformers=[('num',
                                                  Pipeline(steps=[('imputer',
                                                                   SimpleImputer(strategy='median')),
                                                                  ('scaler',
                                                                   StandardScaler())]),
                                                  ['age', 'financial_worry',
                                                   'stress', 'burnout',
                                                   'support', 'satisfaction',
                                                   'gpa', 'sleep_hours',
                                                   'study_hours',
                                                   'exercise_days',
                                                   'caffeine_mg',
                                                   'supervisor_meetings',
                                                   'wellbeing']),
                                                 ('cat',
                                                  Pipeline(steps=[('imputer',
                                                                   SimpleImputer(strategy='most_frequent')),
                                                                  ('ohe',
                                                                   OneHotEncoder(drop='first',
                                                                                 handle_unknown='ignore',
                                                                                 sparse_output=False))]),
                                                  ['gender', 'faculty',
                                                   'programme', 'study_mode',
                                                   'employment', 'has_children',
                                                   'lives_away'])])),
                ('classifier',
                 LogisticRegression(max_iter=1000, random_state=2026))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

The single call to fit() calculates medians and scaling parameters on X_train, encodes categories, and fits the logistic regression model on the transformed data.

11.8 Evaluating a classifier

The model is now judged on the test students. Calling predict() produces class labels (\(0\) or \(1\)), and predict_proba() produces the estimated probabilities:

y_pred_class = logistic_pipe.predict(X_test)
y_pred_prob = logistic_pipe.predict_proba(X_test)[:, 1]

test_results = pd.DataFrame({
    "truth": y_test.values,
    "pred_class": y_pred_class,
    "pred_prob": y_pred_prob
})
print(test_results.head())

   truth  pred_class  pred_prob
0      0           0   0.031820
1      1           1   0.644133
2      0           0   0.177301
3      0           0   0.042008
4      0           0   0.030207

The simplest measure is accuracy: the share of students classified correctly:

acc = accuracy_score(y_test, y_pred_class)
always_no_acc = (y_test == 0).mean()

print(f"Model test accuracy:          {acc:.1%}")
print(f"Always predicting 'No' base: {always_no_acc:.1%}")

Model test accuracy:          88.0%
Always predicting 'No' base: 84.7%

An accuracy of 88.0% sounds good. But a “model” that ignores every predictor and says “No” for every student would be right 84.7% of the time, because most students answer “No”, while finding not one of the students at risk. For an imbalanced outcome, accuracy is a misleading measure.

A better measure is the ROC AUC (the area under the receiver operating characteristic curve). It is the probability that, of one student who considered dropping out and one who did not, the model gives the first a higher predicted probability than the second. An AUC of 0.5 is no better than a coin toss, and 1.0 is perfect. Unlike accuracy, it is not fooled by an imbalanced outcome:

auc_test = roc_auc_score(y_test, y_pred_prob)
print(f"Test ROC AUC: {auc_test:.3f}")

Test ROC AUC: 0.843

An AUC of 0.84 means the model ranks an at-risk student above a not-at-risk student about 84% of the time: a useful model. Chapter 12 looks at classification measures in much more detail, including how to choose the threshold at which a student is flagged.

11.9 Overfitting in practice

The simulation at the start of the chapter showed overfitting with a flexible curve. The same happens with real data. A dramatic example is k-nearest neighbours (k-NN) with \(k = 1\). It predicts each student’s outcome by finding the single most similar student in the training data and copying their answer. On the training data, every student’s nearest neighbour is themselves, so it can never be wrong:

knn1_pipe = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", KNeighborsClassifier(n_neighbors=1))
])
knn1_pipe.fit(X_train, y_train)

knn1_train_auc = roc_auc_score(y_train, knn1_pipe.predict_proba(X_train)[:, 1])
knn1_test_auc = roc_auc_score(y_test, knn1_pipe.predict_proba(X_test)[:, 1])

print(f"KNN (k=1) Training AUC: {knn1_train_auc:.2f}")
print(f"KNN (k=1) Test AUC:     {knn1_test_auc:.2f}")

KNN (k=1) Training AUC: 1.00
KNN (k=1) Test AUC:     0.61

The model scores perfectly on the training data, and much more poorly on new students. The training score was an illusion: the model had memorised the training students, not learned anything that carries over. It is the degree-12 curve again, and the reason every model in this book is tested on data it has not seen.

11.10 Cross-validation

The test set is used only once, at the end. During model building, however, models and settings often need to be compared, and neither the training data (that would reward overfitting) nor the test set (that would use it up) can be used to judge them. Cross-validation solves this by reusing the training data. The training data is split into, say, 10 equal parts, called folds. The model is fitted on 9 folds and its performance measured on the fold left out; this is repeated 10 times, leaving out each fold in turn, and the 10 performance measures are averaged. Every student is used for evaluation exactly once, always by a model that did not see them during fitting. Figure 11.4 shows the idea with 5 folds.

Figure 11.4: Five-fold cross-validation: in each round, the model is fitted on four folds (grey) and evaluated on the fifth (dark blue).

The class StratifiedKFold() creates stratified folds, and cross_validate() fits and evaluates the pipeline across each fold:

cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=2026)
cv_results = cross_validate(logistic_pipe, X_train, y_train, cv=cv, scoring=["roc_auc", "accuracy"])

cv_auc_mean = cv_results["test_roc_auc"].mean()
cv_auc_se = cv_results["test_roc_auc"].std() / np.sqrt(10)

print(f"Cross-validated AUC: {cv_auc_mean:.3f} (SE: {cv_auc_se:.3f})")

Cross-validated AUC: 0.806 (SE: 0.020)

The cross-validated AUC, about 0.81, is an honest estimate of how the model will perform on new students, obtained without touching the test set.

11.11 Tuning a model

Many models have settings that are not learned from the data but must be chosen beforehand, such as \(k\) in k-nearest neighbours or the degree of the polynomials in the simulation. They are called hyperparameters, to distinguish them from the parameters, such as regression coefficients, that the model learns. The best value is found by tuning: trying several values and comparing them with cross-validation.

A decision tree (explained fully in Chapter 12) predicts by asking a series of yes-or-no questions, such as whether stress is above 3.5. Its depth, the number of questions in a row, is a hyperparameter that controls its flexibility. A shallow tree is too simple to capture the patterns and underfits; a very deep tree can fit the training data closely, noise included, and overfits:

tree_pipe = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", DecisionTreeClassifier(random_state=2026, min_samples_leaf=10))
])

param_grid = {"classifier__max_depth": list(range(1, 11))}
grid_search = GridSearchCV(tree_pipe, param_grid=param_grid, cv=cv, scoring="roc_auc")
grid_search.fit(X_train, y_train)

tuning_results = pd.DataFrame({
    "max_depth": range(1, 11),
    "mean_auc": grid_search.cv_results_["mean_test_score"]
})
print(tuning_results.head(6).round(3))

   max_depth  mean_auc
0          1     0.634
1          2     0.761
2          3     0.764
3          4     0.739
4          5     0.687
5          6     0.675

fig, ax = plt.subplots(figsize=(6, 3.6))
ax.plot(tuning_results["max_depth"], tuning_results["mean_auc"], marker="o", color=navy, linewidth=1.5)
ax.set_xticks(range(1, 11))
ax.set_xlabel("Tree depth (max_depth)")
ax.set_ylabel("Cross-validated ROC AUC")
ax.set_title("Hyperparameter Tuning: Decision Tree Depth")
plt.tight_layout()
plt.show()

Figure 11.5: Cross-validated ROC AUC of decision trees of different depths.

best_depth = grid_search.best_params_["classifier__max_depth"]
print("Best max_depth:", best_depth)
print(f"Best CV ROC AUC: {grid_search.best_score_:.3f}")

Best max_depth: 3
Best CV ROC AUC: 0.764

The pattern is the new-data curve of Figure 11.2 seen from the other side: a tree of depth 1 is no better than chance, the AUC rises as the tree grows, and beyond a moderate depth it levels off and dips slightly as the deeper trees start to overfit. Here, GridSearchCV selects max_depth = 3.

11.12 The final test

Only now, with the model chosen, is the test set used. The tuned decision tree is evaluated on the test set, alongside plain logistic regression:

best_tree = grid_search.best_estimator_
tree_test_auc = roc_auc_score(y_test, best_tree.predict_proba(X_test)[:, 1])
tree_test_acc = accuracy_score(y_test, best_tree.predict(X_test))

print(f"Tuned Tree Test AUC:        {tree_test_auc:.3f} (Accuracy: {tree_test_acc:.1%})")
print(f"Logistic Regression Test AUC: {auc_test:.3f} (Accuracy: {acc:.1%})")

Tuned Tree Test AUC:        0.725 (Accuracy: 84.7%)
Logistic Regression Test AUC: 0.843 (Accuracy: 88.0%)

The tuned tree reaches an AUC of 0.73; plain logistic regression, 0.84. The more flexible model is not the better one. In this data, the risk of dropout rises steadily with stress and falls steadily with support, so a simple smooth model captures the pattern well, while a tree, which splits the data into rectangular boxes, captures it less well. Trying a simple model first, and making a complex one earn its place, is one of the most useful habits in machine learning.

WarningPredictions about people

A model that flags students “at risk” affects real people. Before such a model is used, the consequences of its errors must be considered: a student flagged by mistake may be treated differently, and a student who is missed receives no help. The model should be checked separately for different groups, such as women and men or part-time and full-time students, because a model with a good AUC overall can still be unfair to a particular group. And its purpose matters: in this study, the right use is to offer support earlier, never to penalise.

NoteIn your field: business and finance

Banks use exactly these methods to predict which loan applicants will repay. The credit_data dataset records real loan applications, with whether each loan turned out well or badly (Status). The same workflow applies:

credit_clean = credit_data.dropna(subset=["Status"]).copy()
X_cr = credit_clean.drop(columns=["Status"])
y_cr = (credit_clean["Status"] == "bad").astype(int)

X_train_c, X_test_c, y_train_c, y_test_c = train_test_split(
    X_cr, y_cr, test_size=0.25, stratify=y_cr, random_state=1
)

num_c = X_cr.select_dtypes(include=[np.number]).columns.tolist()
cat_c = X_cr.select_dtypes(exclude=[np.number]).columns.tolist()

credit_pre = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), num_c),
    ("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")), ("ohe", OneHotEncoder(drop="first", sparse_output=False, handle_unknown="ignore"))]), cat_c)
])

credit_pipe = Pipeline([
    ("preprocessor", credit_pre),
    ("classifier", LogisticRegression(max_iter=1000, random_state=1))
])
credit_pipe.fit(X_train_c, y_train_c)

c_prob = credit_pipe.predict_proba(X_test_c)[:, 1]
c_pred = credit_pipe.predict(X_test_c)

print(f"Credit Test ROC AUC:    {roc_auc_score(y_test_c, c_prob):.3f}")
print(f"Credit Test Accuracy:   {accuracy_score(y_test_c, c_pred):.1%}")
print(f"Credit Brier Score:     {brier_score_loss(y_test_c, c_prob):.3f}")

Credit Test ROC AUC:    0.814
Credit Test Accuracy:   79.1%
Credit Brier Score:     0.149

Only the data and the outcome have changed; every step of the workflow is the same. The output also shows the Brier score, another measure of probability accuracy: 0 is perfect, and lower is better.

11.13 Common misconceptions

Predictive modelling has its own characteristic mistakes, most of them versions of testing a model on what it already knows.

• “A model that fits the training data well will predict well.” Training performance rewards overfitting; only performance on new data counts.
• “A more complex model is a better model.” Beyond a point, flexibility adds variance, not accuracy. Simple models often predict as well.
• “High accuracy means a good classifier.” For an imbalanced outcome, a model that predicts the common class every time can have high accuracy and no value.
• “The test set can be used to choose between models.” Every use of the test set for a decision makes it less of a test. Choices are made with cross-validation; the test set is used once.
• “A good predictor is a cause.” Prediction needs association, not causation; a predictive model says nothing about what would happen if a predictor were changed.

11.14 Chapter review

11.14.1 Summary

• Supervised learning predicts an outcome (classification for categories, regression for numbers); unsupervised learning finds structure. Prediction is a different research aim from explanation, judged by performance on new data.
• A model generalises when it learns the pattern and not the noise. Training error keeps falling as models become more flexible, while error on new data falls and then rises: the bias-variance trade-off.
• scikit-learn designs a fair test: it splits data (train_test_split), prepares it (ColumnTransformer), specifies models and pipelines (Pipeline), tunes them (GridSearchCV), and measures performance (metrics).
• Split the data into training and test sets, stratified by the outcome, and use the test set only once, at the end. Use only information available at the time of prediction.
• A preprocessor prepares data (imputation, dummy variables, normalisation) using the training data only, so no information leaks from the test set.
• Accuracy misleads for imbalanced outcomes; the ROC AUC is a better summary.
• Cross-validation estimates performance honestly without using the test set. Hyperparameters are chosen by tuning with cross-validation. Simple models often predict as well as complex ones; try them first.
• Predictions about people raise questions of fairness and use.

11.14.2 Key terms

Machine learning, supervised learning, unsupervised learning, classification, regression, prediction, explanation, generalisation, overfitting, underfitting, bias-variance trade-off, imbalanced outcome, training set, test set, stratified split, ColumnTransformer, SimpleImputer, OneHotEncoder, StandardScaler, data leakage, Pipeline, accuracy, ROC AUC, k-nearest neighbours, cross-validation, fold, StratifiedKFold, hyperparameter, parameter, tuning, GridSearchCV, decision tree, Brier score.

11.15 Exercises

The playground has these and more, with hints and solutions.

1. Remove the questionnaire scores (stress, burnout, support, satisfaction) from dropout_data and fit the logistic regression pipeline again. Report how much the cross-validated AUC falls, and what that says about the questionnaire.
2. Split the data 80/20 instead of 75/25 with a different random state (e.g. random_state=7), and describe how much the test AUC of the logistic model changes, and why it might.
3. Tune k-nearest neighbours over n_neighbors = [5, 11, 21, 41] with cross-validation. Identify the best value, and compare its AUC with logistic regression.
4. Add zero-variance filtering (VarianceThreshold()) to the preprocessing steps. Look up what “zero variance” means, and explain when this step is useful.
5. In two or three sentences, explain to a university administrator why a model with 85% accuracy might still be useless for finding students at risk.
6. Repeat the polynomial simulation with 50 training points instead of 15. Describe how the error curve on new data changes, and explain why more data allows a more flexible model.

11.16 Further reading

• An Introduction to Statistical Learning: with Applications in Python (James et al. 2023) explains the ideas behind overfitting, the bias-variance trade-off, cross-validation, and machine learning methods.
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron (Géron 2022) is the definitive guide to scikit-learn pipelines and predictive modeling.

References

Géron, Aurélien. 2022. Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow. 3rd ed. O’Reilly Media.

James, Gareth, Daniela Witten, Trevor Hastie, Robert Tibshirani, and Jonathan Taylor. 2023. An Introduction to Statistical Learning: With Applications in Python. Springer. https://doi.org/10.1007/978-3-031-38747-0.