Playground: Chapter 8

ANOVA and Regression

This page practises the ideas of Chapter 8 in four parts. Practise the chapter works through the book’s exercises, with starter code, hints, and solutions. Go further goes beyond the book, with new questions and real-world datasets. Check your understanding asks short questions whose answers open with a click. Do it yourself gives open tasks with an empty code space and no answers: you build and check the models yourself, as you will in your own thesis.

In the exercises, replace each ______ with your own code and press Run Code. The DataFrame study is prepared as in the chapter: one row per student, with background variables, stress and support scores, and first-semester records.

Practise the chapter

These exercises follow the book’s exercises closely.

Exercise 1: Wellbeing across faculties

Test whether first-semester wellbeing differs between faculties with a one-way ANOVA using smf.ols and anova_lm, and calculate \(\eta^2\). If the ANOVA is significant, run Tukey’s test; if not, explain why a post-hoc test is not needed.

Solution

wellbeing_anova = smf.ols("wellbeing ~ C(faculty)", data=study).fit()
aov_table = sm.stats.anova_lm(wellbeing_anova, typ=1)
print(aov_table)

ss_between = aov_table.loc["C(faculty)", "sum_sq"]
ss_total = aov_table["sum_sq"].sum()
eta_sq = ss_between / ss_total
print(f"Eta squared: {eta_sq:.4f}")

The ANOVA gives \(F(4, 595) = 2.02, p = 0.09\), with an \(\eta^2\) of about 0.013 (just 1.3% of the variation in wellbeing sits between faculties). Because the p-value is above 0.05, there is no evidence that faculties differ in wellbeing. A post-hoc test is not needed: if the overall F test finds no evidence of differences, testing individual pairs is fishing for false alarms.

Exercise 2: Wellbeing and sleep

Fit a simple regression of first-semester wellbeing on sleep hours, and interpret the slope and R-squared.

Solution

wb_simple = smf.ols("wellbeing ~ sleep_hours", data=study).fit()
print(wb_simple.summary())

The slope is 5.30 (\(p < .001\)): each extra hour of sleep per night is associated with a wellbeing score about 5.3 points higher on the 0-to-100 scale. \(R^2 = 0.208\): sleep alone explains about 21% of the differences in first-semester wellbeing.

Exercise 3: Adding stress and support

Add stress and supervisor support to the model from Exercise 2. Describe what happens to the slope of sleep and to R-squared.

Solution

wb_simple = smf.ols("wellbeing ~ sleep_hours", data=study).fit()
wb_multiple = smf.ols("wellbeing ~ sleep_hours + stress + support", data=study).fit()

print("Simple slope for sleep:  ", wb_simple.params["sleep_hours"])
print("Multiple slope for sleep:", wb_multiple.params["sleep_hours"])
print("Simple R-squared:        ", wb_simple.rsquared)
print("Multiple R-squared:      ", wb_multiple.rsquared)

The slope of sleep drops from 5.30 to 3.90, because students who sleep more also experience less stress and report better support. Once stress and support are held constant, sleep’s unique association is somewhat smaller, but remains strongly significant (\(p < .001\)). Meanwhile, \(R^2\) doubles from 0.21 to 0.43: together, the three predictors explain 43% of the variation in wellbeing.

Exercise 4: A categorical predictor

Add study_mode to the multiple regression model from Exercise 3. Identify the reference category and explain its coefficient. Then fit the same model with Part-time as the reference category.

Solution

wb_mode = smf.ols("wellbeing ~ sleep_hours + stress + support + C(study_mode)", data=study).fit()
print(wb_mode.summary().tables[1])

wb_mode_ref = smf.ols(
    "wellbeing ~ sleep_hours + stress + support + C(study_mode, Treatment(reference='Part-time'))",
    data=study
).fit()
print(wb_mode_ref.summary().tables[1])

The default reference category is Full-time (alphabetical first), so the coefficient for Part-time is −3.01 points (\(p < .001\)): part-time students report wellbeing 3 points lower than full-time students with the same sleep, stress, and support. With Part-time as the reference, Full-time is +3.01 points. The choice of reference category changes the perspective of the table, never the underlying finding.

Exercise 5: Burnout and dropout

Fit a logistic regression of considering dropout on the burnout score alone, calculating the composite score from the questionnaire first. Report its odds ratio and find the predicted probability of considering dropout for students with burnout scores of 2 and 4.

Solution

q_burnout = questionnaire.filter(like="burnout_").mean(axis=1)
study_b = study.copy()
study_b["burnout"] = q_burnout
study_b["dropout"] = (study_b["considering_dropout"] == "Yes").astype(int)

burnout_model = smf.logit("dropout ~ burnout", data=study_b).fit(disp=False)
print("Odds ratio:\n", np.exp(burnout_model.params))
print("95% CI of Odds Ratio:\n", np.exp(burnout_model.conf_int()))

new_cases = pd.DataFrame({"burnout": [2.0, 4.0]})
probs = burnout_model.predict(new_cases)
print("Predicted probabilities:", probs.round(3).tolist())

The odds ratio for burnout is 3.69 (95% CI [2.55, 5.36]): each additional point of burnout multiplies the odds of considering dropout by 3.7. The predicted probability for a student with low burnout (score 2) is only 3.1%, whereas for a student with high burnout (score 4) it rises to 30.7%.

Exercise 6: A confounder in your own field

Draw a diagram, like the chapter’s diagram of caffeine, sleep, and GPA, of a confounder that might explain an association in your own field, and describe how a regression could separate the two explanations. Write your answer first, then open the model answer.

Model answer

An example from public health: towns with more fast-food outlets have higher rates of heart disease. Explanation 1: fast food causes heart disease (outlets → disease). Explanation 2: poverty is a common cause (poverty → outlets, poverty → disease), and the outlets themselves have no effect. The diagram has three boxes with arrows from poverty to both outlets and disease, and a question-marked arrow from outlets to disease. A regression of disease rate on the number of outlets and a measure of poverty (such as average income) separates the two: if the coefficient of outlets stays clearly above zero with poverty in the model, explanation 1 survives; if it shrinks towards zero, poverty was confounding the association. The regression can only allow for confounders that were measured; an unmeasured one, such as smoking rates, could still explain what remains.

Go further

These exercises explore deeper statistical mechanics.

Exercise 7: F by hand

Build the F statistic for the chapter’s three small groups of stress scores step by step, and compare it with anova_lm.

Solution

F_value = (ss_between / 2) / (ss_within / 9)
print("Manual F value:", F_value)

The between-group sum of squares is 1.31, the within-group sum of squares is 0.55, and \(F = (1.307 / 2) / (0.550 / 9) = 10.69\), which exactly matches anova_lm() (\(p = 0.004\)).

Exercise 8: Regression as conditional means

Group students by their sleep rounded to the nearest whole hour, calculate average wellbeing in each group, and compare the empirical averages with the regression line’s predictions.

Solution

bands["line_prediction"] = wb_line.predict(pd.DataFrame({"sleep_hours": bands["sleep_band"]}))
print(bands)

Where most students are, the line and empirical averages agree within a single point (e.g., at 6 hours: 58.2 vs 57.9; at 7 hours: 63.7 vs 63.2). At the sparsely populated tails (7 students at 9 hours, 3 at 10 hours), empirical averages fluctuate noisily around the regression line.

Exercise 9: Simpson’s paradox

The classic UCBAdmissions data records graduate applications to the six largest departments of UC Berkeley in 1973. Calculate the overall admission rate for men and women, then calculate the rate within each department.

Solution

overall["rate"] = overall["admitted"] / overall["total"]
print("Overall admission rates:\n", overall)

Overall, 44.5% of male applicants and 30.4% of female applicants were admitted. However, within individual departments, women had a higher admission rate in four of the six departments (e.g. Dept A: 82.4% vs 62.1%; Dept B: 68.0% vs 63.0%). Department was a major confounder: female applicants applied disproportionately to departments with low overall admission rates.

Exercise 10: A missed curve in the residuals

Compare a straight linear regression of GPA on study hours with a quadratic model using np.power. Compare their \(R^2\) values.

Solution

straight = smf.ols("gpa ~ study_hours", data=study).fit()
curved = smf.ols("gpa ~ study_hours + np.power(study_hours, 2)", data=study).fit()

print(f"Straight R-squared: {straight.rsquared:.4f}")
print(f"Curved R-squared:   {curved.rsquared:.4f}")

The straight line explains almost none of the variance (\(R^2 = 0.004\)), whereas the curved model explains 3.6% (\(R^2 = 0.036\)). The quadratic model captures diminishing returns, where study benefits peak around 32 hours per week.

Check your understanding

Try answering these conceptual questions in your own words before opening the model answers.

1. What does the F statistic of an ANOVA compare?

The F statistic compares the variation between group averages against the variation within groups, each divided by its degrees of freedom. An F close to 1 means the group averages differ no more than chance would produce; a large F indicates that group differences exceed chance expectations.

2. In a multiple regression, what does “holding the other predictors constant” mean?

It means comparing cases that differ by one unit on that predictor while sharing identical values for all other predictors in the model. It is a statistical adjustment that removes confounding from measured variables.

3. What is a confounder?

A confounder is a third variable that is a common cause of both the predictor and the outcome. If unmeasured or unadjusted, it creates or distorts the observed association between predictor and outcome.

4. A residual plot shows a funnel pattern. What does it mean?

It indicates heteroscedasticity: the variance of the residuals is not constant across fitted values. While the estimated regression slopes remain unbiased, standard errors, p-values, and confidence intervals are unreliable without robust standard errors or a variance-stabilising transformation.

5. An odds ratio of 2 does not mean that the probability doubles. Why not?

Odds are defined as \(p / (1 - p)\). For very rare events (\(p \approx 0.01\)), doubling the odds approximately doubles the probability. But for an event with 50% probability (odds = 1.0), doubling the odds to 2.0 corresponds to a probability of \(2 / (1 + 2) \approx 67\%\), not 100%.

Do it yourself

These open tasks require you to choose variables and write code from scratch, as you will in your thesis.

Task 1: Wellbeing regression and reporting

Fit a regression model for first-semester wellbeing with two or three predictors of your choice. Inspect the diagnostic residual plot, and write a thesis-style reporting paragraph with every number calculated by code.

Task 2: Interaction between support and study mode

Does the effect of supervisor support on wellbeing differ between full-time and part-time students? Centre support, fit an interaction model, and draw an interaction plot.

Task 3: Logistic regression for living away from family

Fit a logistic regression predicting lives_away with two or three background variables. Report odds ratios with 95% confidence intervals, and predict probabilities for two contrasting profiles.

Task 4: Modeling timber volume in the cherry trees dataset

Using trees.csv, model Volume from Girth and Height. Check whether a linear model is adequate or if a transformation/polynomial is required.