Playground: Chapter 15
Neural Networks
This page practises the ideas of Chapter 15 in four parts. Practise the chapter works through the book’s exercises, with starter code, hints, and solutions. Go further goes beyond the book, with new questions and real datasets. Check your understanding asks short questions whose answers open with a click. Do it yourself gives open tasks with an empty code space and no answers: you decide whether a network is worth it, as you will in your own thesis.
In the exercises below, replace each ______ with your own code and press Run Code.
Practise the chapter
These are the exercises at the end of Chapter 15, with the same numbers. In the setup, train and test hold the students, with dropout (1 for “Yes”) and nine normalised predictors; sigmoid() is defined as in the chapter.
Exercise 1: A neuron with new weights
Change the neuron’s weights so that support matters twice as much as in the chapter, recalculate the outputs for the two students (stress 4 and support 2; stress 2 and support 4), and explain what a negative weight means.
The chapter’s weight for support was −0.8; twice as much, in the same direction, is twice that number (−1.6).
def neuron(stress, support):
return sigmoid(-4 + 1.2 * stress - 1.6 * support)The high-stress, low-support student’s output falls from 0.31 to 0.08, and the low-stress, high-support student’s from 0.008 to 0.0003. A negative weight means that the input lowers the output: every extra point of support pushes the total, and so the predicted risk, down. Doubling the weight doubles how strongly support pulls the total down, but because the sigmoid is curved, the effect on the probability depends on where the total starts.
Exercise 3: When the seed matters
Fit a network with 5 hidden neurons three times with different random seeds, with a weight decay (alpha) of 1.0, then do the same with an alpha of 0.01 and with none (alpha=0). Explain when the seed matters, and why.
Pass the loop variable seed into random_state=seed to test different initial weight configurations.
net = MLPClassifier(hidden_layer_sizes=(5,), alpha=alpha_val,
max_iter=500, random_state=seed, solver='lbfgs')With an alpha of 1.0, the three seeds give consistent, stable test AUCs. With an alpha of 0.01 or 0, they fluctuate considerably. Training starts from random weights, and with little or no penalty, a network with 5 hidden neurons has many different sets of weights that fit the training data about equally well, so each seed ends somewhere different, and each overfits in its own way. A strong penalty pulls every run towards the same small-weight solution. If results change with the seed, the model is not regularised enough, and the seed must be reported.
Exercise 4: Epochs
Train a 20-neuron network without weight decay for 10, 100, and 1,000 epochs (max_iter), and describe how the training and test AUCs change.
Pass max_iter=epochs inside the loop.
net = MLPClassifier(hidden_layer_sizes=(20,), alpha=0.0,
max_iter=epochs, random_state=1, solver='lbfgs')After 10 epochs, the network learns the broad pattern. As iterations continue to 100 and 1000, the training AUC climbs to a perfect 1.000 (memorisation), while the test AUC plateaus or deteriorates. The network has far more weights than necessary for the sample size, fitting the training noise.
Exercise 5: Explaining the choice to the research group
In two or three sentences, explain to the research group why the thesis does not use a neural network for the dropout model. Write your answer first, then open the model answer.
“A carefully tuned neural network predicted dropout no better than logistic regression: both reached a cross-validated AUC of about 0.81 to 0.83, and the small difference on the test students is within chance. The network’s hundreds of weights cannot be interpreted, while logistic regression gives odds ratios that show how each factor relates to the risk of dropping out, so we kept logistic regression and report the network as a model we tried.”
Exercise 6: The learning rate
Repeat the chapter’s gradient-descent loop with a learning rate of 0.1 and of 10, and then of 100. Describe how the loss curve changes, and explain what goes wrong when the learning rate is too large.
The blank is the middle learning rate from the exercise (10.0).
for lr in [0.1, 10.0, 100.0]:With 0.1, the steps are small: the loss falls smoothly but slowly, still above 0.40 after 100 steps. With 10, the step overshoots initially but settles quickly. With 100, each step overshoots so wildly that the weights oscillate uncontrollably, and numerical instability ensues. A learning rate that is too large makes gradient descent jump across the minimum rather than descending into it.
Go further
These exercises go beyond the book.
Exercise 7: Gradient descent with two predictors
Train a neuron with two inputs, stress and support, by gradient descent on the training students, and compare its weights with LogisticRegression.
The prediction error is p - y_vec.
grad = X_mat.T @ (p - y_vec) / len(y_vec)After 2,000 steps, the gradient descent weights match the exact maximum likelihood coefficients from logistic regression to four decimal places.
Exercise 9: Weight decay with a small sample
Train networks with 5 hidden neurons on only 60 training students, with weight decays (alpha) of 0, 0.1, 1.0, and 10.0 and three random seeds each, and compare them with logistic regression.
Pass alpha=alpha_val into MLPClassifier.
net = MLPClassifier(hidden_layer_sizes=(5,), alpha=alpha_val,
max_iter=1000, random_state=seed, solver='lbfgs')Without weight decay, the test AUC varies widely across seeds due to overfitting. A strong penalty stabilizes the solution.
Exercise 10: A network for timber volume
The trees.csv dataset records the girth, height, and timber volume of 31 black cherry trees. Fit a regression network (MLPRegressor) and a linear regression, and compare them with leave-one-out cross-validation.
The straight line predicts a left-out tree’s volume with an RMSE of ~4.3 cubic feet; the non-linear neural network achieves an RMSE of ~3.4 cubic feet, capturing the physical allometric curvature of tree volume as a function of girth and height!
Check your understanding
Answer each question in your own words first, then click to see a model answer.
1. Why is a single neuron with a sigmoid activation the same as a logistic regression?
Because it computes the same thing: a weighted sum of the inputs plus a constant, passed through the sigmoid, which turns any number into a probability. The bias is the intercept, the weights are the coefficients, and training it by minimising the log loss finds the same values as standard logistic regression. Networks become more than logistic regression only when neurons are combined in layers.
2. What does the learning rate control?
How far the weights move at each step of gradient descent, in the direction that reduces the error. Too small, and training is slow; too large, and each step overshoots the minimum, so the error bounces around or grows without limit. It is a setting chosen before training, not something the network learns.
3. What does weight decay prevent?
Overfitting. It adds a penalty on large weights to the error that training minimises, like the ridge penalty in Chapter 13. Without it, a network with many weights can keep adjusting them until it memorises the noise of its training data; with it, the weights stay small, the network’s function stays smooth, and different random starts end at similar solutions.
4. Why must the seed be set, and reported, when training a neural network?
Because training starts from random weights, and a network can end at different solutions depending on where it starts, especially with little regularisation. Setting the seed makes the result reproducible; reporting it lets others repeat it. If results change noticeably with the seed, that is itself a warning that the model is not stable enough to trust.
5. When are neural networks worth using?
When there is a lot of data and the relationship is complex, with many interacting inputs whose combinations matter, as in images, sound, and text, where deep networks learn features that no one could specify by hand. For tabular data of a few hundred cases with a smooth relationship, like most thesis data, they rarely beat regression, and they cost interpretability, stability, and tuning effort.
Do it yourself
These tasks have no starter code and no answers. Each code space below is empty and ready to run: write your own code, as you would for a thesis.
1. Build a neural network for a numeric outcome, such as semester 1 GPA or wellbeing, using MLPRegressor. Choose the number of hidden neurons and the weight decay (alpha) by cross-validation, compare the network fairly with a linear model on the same folds, and decide which you would use.
2. Explain to a friend who has never studied statistics what “training” a neural network means, using your own gradient-descent loop as the example: what the loop starts from, what each step does, and how it knows when it is done. Write the explanation as comments above the code.
3. For each of these three research projects, decide whether a neural network is worth trying, and why: (a) predicting the diagnosis of a skin lesion from 20,000 labelled photographs; (b) predicting which of 300 survey respondents will change jobs, from 12 questionnaire scores; (c) predicting next week’s hospital admissions from ten years of daily records. No code is needed; write your answers on paper or in a document.
Work on your own computer
The project contains the data and all four parts of this page as a Python script (exercises.py). Answers to the questions are in solutions.py, and the open tasks have space to write your code.
- Download chapter15.zip, unzip it, and open it in your favourite Python IDE.