Playground: Chapter 7
Hypothesis Testing and Statistical Inference
This page practises the ideas of Chapter 7 in four parts. Practise the chapter works through the book’s exercises, with starter code, hints, and solutions. Go further goes beyond the book, with new questions and statistical simulations. Check your understanding asks short conceptual questions whose answers open with a click. Do it yourself gives open tasks with an empty code space and no answers: you decide how to analyse the data, as you will in your own thesis. Several results on this page are not significant; a non-significant result is still a result.
In the exercises, replace each ______ with your own code and press Run Code.
Practise the chapter
These are the exercises at the end of Chapter 7, with the same numbers.
Exercise 1: Two sampling distributions
Draw 1,000 samples of 10 students’ first-semester study hours, and 1,000 samples of 50. Compare the spread (SD) of the two sets of averages, plot the averages of 50, and explain the difference.
The second set of samples draws 50 students (size=50).
study = first_sem["study_hours"].dropna().to_numpy()
rng = np.random.default_rng(1)
means_10 = np.array([rng.choice(study, size=10, replace=True).mean() for _ in range(1000)])
means_50 = np.array([rng.choice(study, size=50, replace=True).mean() for _ in range(1000)])
print(f"SE for n=10: {means_10.std(ddof=1):.2f} hours")
print(f"SE for n=50: {means_50.std(ddof=1):.2f} hours")
fig, ax = plt.subplots(figsize=(6, 3.5))
sns.histplot(means_50, bins=30, color="#1d588c", ax=ax)
ax.set_title("Sampling Distribution of Mean Study Hours (n=50)")
ax.set_xlabel("Sample Mean (hours/week)")
plt.show()The sample averages of 50 students vary far less (SE \(\approx 1.85\) hours, compared to \(\approx 4.13\) hours for samples of 10). Larger samples provide much more precise estimates: standard error shrinks with \(1/\sqrt{n}\), so 5 times as many students yields an error \(\sqrt{5} \approx 2.24\) times smaller.
Exercise 2: A confidence interval for GPA
Calculate a 95% confidence interval for average first-semester GPA, and interpret what it tells you.
In SciPy, stats.ttest_1samp computes Student’s \(t\)-test and provides .confidence_interval(0.95).
gpa_clean = first_sem["gpa"].dropna()
res_gpa = stats.ttest_1samp(gpa_clean, popmean=0)
ci_gpa = res_gpa.confidence_interval(0.95)
print(f"Mean GPA: {gpa_clean.mean():.2f}")
print(f"95% CI: [{ci_gpa.low:.2f}, {ci_gpa.high:.2f}]")The sample mean GPA is 3.19. The 95% confidence interval is [3.15, 3.23]. In words: the true mean first-semester GPA of all graduate students is plausibly bracketed between 3.15 and 3.23 grade points.
Exercise 3: A one-sample test against 60
Test whether average first-semester student wellbeing differs from 60. State the hypotheses, run the test, and write a conclusion.
The hypothesized population mean under \(H_0\) is popmean=60.
test_wb = stats.ttest_1samp(first_sem["wellbeing"].dropna(), popmean=60)
ci_wb = test_wb.confidence_interval(0.95)
print(f"t-statistic: {test_wb.statistic:.2f}")
print(f"df: {test_wb.df}")
print(f"p-value: {test_wb.pvalue:.4e}")
print(f"95% CI: [{ci_wb.low:.2f}, {ci_wb.high:.2f}]")The sample mean is 52.88 (\(SD = 11.23\)). The \(t\)-statistic is \(-15.54\), and the \(p\)-value is \(< .001\). First-semester graduate student wellbeing is significantly below the neutral midpoint of 60 (\(t(599) = -15.54, p < .001, 95\%\text{ CI }[51.98, 53.78]\)).
Exercise 4: Full-time and part-time students
Compare the first-semester GPA of full-time and part-time students, first with a permutation test of 5,000 shuffles, then with a Welch two-sample \(t\)-test. Calculate Cohen’s \(d\).
Use rng.permutation(all_gpa) to shuffle array elements. For Welch’s \(t\)-test in SciPy, specify equal_var=False.
ft_gpa = first_sem.query("study_mode == 'Full-time'")["gpa"].dropna().to_numpy()
pt_gpa = first_sem.query("study_mode == 'Part-time'")["gpa"].dropna().to_numpy()
observed_diff = ft_gpa.mean() - pt_gpa.mean()
all_gpa = np.concatenate([ft_gpa, pt_gpa])
n_ft = len(ft_gpa)
rng = np.random.default_rng(2026)
shuffled_diffs = []
for _ in range(5000):
perm = rng.permutation(all_gpa)
shuffled_diffs.append(perm[:n_ft].mean() - perm[n_ft:].mean())
perm_p = (np.abs(shuffled_diffs) >= np.abs(observed_diff)).mean()
welch_res = stats.ttest_ind(ft_gpa, pt_gpa, equal_var=False)
s_pooled = np.sqrt((ft_gpa.var(ddof=1) + pt_gpa.var(ddof=1)) / 2)
cohens_d = (ft_gpa.mean() - pt_gpa.mean()) / s_pooled
print(f"Observed Difference: {observed_diff:.3f}")
print(f"Permutation p-value: {perm_p:.3f}")
print(f"Welch t p-value: {welch_res.pvalue:.3f}")
print(f"Cohen's d: {cohens_d:.3f}")Full-time and part-time students differ by only 0.03 grade points. Over 44% of shuffles produce a gap at least that large, matching Welch’s \(t\)-test (\(p = 0.44, d = 0.06\)). Study mode does not create differences in student GPA.
Exercise 5: Power by simulation
Simulate the power of a study with 40 students per group when the true workshop effect is 3 points (\(SD = 11\)).
Pass 40 as the first argument, and effect=3.0.
power_40 = simulate_study(40, effect=3.0)
print(f"Power with 40 students per group: {power_40:.2f}")Power is only ~0.22 (22%). A study with 40 students per group has an intolerable 78% risk of committing a Type II error (failing to detect the real 3-point intervention effect).
Exercise 6: Living away and dropout
Test whether living away from family (lives_away) is related to considering dropout. Create the contingency table, run a chi-square test, and inspect expected frequencies.
Pass cont_tab directly to scipy.stats.chi2_contingency().
cont_tab = pd.crosstab(students["lives_away"], students["considering_dropout"])
print("Contingency table:")
print(cont_tab)
chi2_out = stats.chi2_contingency(cont_tab)
print(f"Chi-square: {chi2_out.statistic:.2f}, df: {chi2_out.dof}, p-value: {chi2_out.pvalue:.4f}")
print("Expected frequencies:")
print(np.round(chi2_out.expected_freq, 1))All expected counts are well above 40. The chi-square test indicates no statistically significant association (\(\chi^2(1) = 2.45, p = 0.117\)).
Go further
These exercises go beyond the book.
Exercise 7: Confidence interval coverage
Check that 95% of calculated confidence intervals truly contain the population mean: draw 100 samples of 30 from the first-semester wellbeing scores, compute 95% intervals, and count coverage.
Specify confidence_level=0.95 inside .confidence_interval(0.95).
pop = first_sem["wellbeing"].to_numpy()
true_param = pop.mean()
rng = np.random.default_rng(4)
covered_count = 0
for _ in range(100):
sample_draw = rng.choice(pop, size=30, replace=False)
ci = stats.ttest_1samp(sample_draw, popmean=0).confidence_interval(0.95)
if ci.low <= true_param <= ci.high:
covered_count += 1
print(f"Coverage out of 100 samples: {covered_count}%")Exactly 95 of the 100 intervals contain the true population parameter.
Exercise 8: Permutation test for medians
Test whether median sleep differs between female and male students using 5,000 shuffles of the gender labels.
Compare the observed median difference against the shuffled distribution.
fem_sleep = first_sem.query("gender == 'Female'")["sleep_hours"].dropna().to_numpy()
male_sleep = first_sem.query("gender == 'Male'")["sleep_hours"].dropna().to_numpy()
obs_median_diff = np.median(fem_sleep) - np.median(male_sleep)
all_sleep = np.concatenate([fem_sleep, male_sleep])
n_f = len(fem_sleep)
rng = np.random.default_rng(2026)
shuffled_median_diffs = []
for _ in range(5000):
perm = rng.permutation(all_sleep)
shuffled_median_diffs.append(np.median(perm[:n_f]) - np.median(perm[n_f:]))
perm_median_p = (np.abs(shuffled_median_diffs) >= np.abs(obs_median_diff)).mean()
print(f"Observed Median Difference: {obs_median_diff:.2f} hours")
print(f"Permutation p-value: {perm_median_p:.3f}")The medians differ by only 0.10 hours (\(p \approx 0.81\)): there is no gender difference in sleep duration.
Exercise 9: Multiple testing correction
Compare wellbeing between all 5 faculties pairwise. Apply a Bonferroni correction.
Specify method="bonferroni".
reject, p_bonf, _, _ = multipletests(p_raw, alpha=0.05, method="bonferroni")
print(f"Significant uncorrected: {sum(np.array(p_raw) < 0.05)} of {len(p_raw)}")
print(f"Significant Bonferroni: {sum(reject)} of {len(p_raw)}")Without correction, 2 of 10 pairwise comparisons appear “significant” at \(p < .05\). After Bonferroni correction, zero remain significant, illustrating how family-wise error inflation creates spurious discoveries.
Exercise 10: Paired vs independent t-test on Student’s sleep data
Compare the extra hours of sleep gained by 10 patients across two soporific drugs using both paired and independent tests.
Notice how ttest_rel controls for baseline patient variance.
paired_result = stats.ttest_rel(d2, d1)
independent_result = stats.ttest_ind(d2, d1, equal_var=False)
print(f"Paired t-test: t = {paired_result.statistic:.2f}, p = {paired_result.pvalue:.4f}")
print(f"Independent t-test: t = {independent_result.statistic:.2f}, p = {independent_result.pvalue:.4f}")The paired test demonstrates a significant difference (\(t = 4.06, p = .0028\)), while the independent test fails to reach significance (\(p = .079\)). Pairing cancels out large between-patient baseline differences.
Exercise 11: Bootstrap and rank tests
Draw a sample of 50 students’ caffeine intake and calculate a 95% bootstrap confidence interval for its median with 2,000 resamples. Then compare full-time vs part-time students with a Mann-Whitney U test.
Draw bootstrap resamples with replace=True.
boot_ci = [np.percentile(boot_meds, 2.5), np.percentile(boot_meds, 97.5)]
print(f"Sample Median: {np.median(small):.1f} mg")
print(f"95% Bootstrap CI: [{boot_ci[0]:.1f}, {boot_ci[1]:.1f}] mg")
print(f"Mann-Whitney U: {u_res.statistic:.1f}, p-value: {u_res.pvalue:.3f}")The sample median is 120.0 mg with 95% bootstrap interval [100.0, 150.0] mg. The Mann-Whitney U test indicates no significant difference between full-time and part-time students (\(p = 0.50\)).
Check your understanding
Answer each question in your own words first, then click to see a model answer.
1. A study reports \(p = 0.03\) for the difference between two groups. What does this number mean, and what does it not mean?
Under the assumption that there is truly zero difference between the groups in the population, a difference at least as large as the observed difference would emerge in approximately 3% of repeated studies. It is not the probability that the null hypothesis is true, and not the probability that the finding occurred by chance.
2. A study of 12 students finds no significant difference between two teaching methods. Can it conclude that the methods are equally effective?
No. With only 12 students, the study suffers from severe statistical underpowering, meaning it would miss even substantial true effects. “Absence of evidence is not evidence of absence.”
3. What is the difference between the standard deviation of a variable and the standard error of its mean?
The standard deviation (\(s\)) quantifies how much individual data points vary around the sample mean. The standard error (\(SE = s / \sqrt{n}\)) quantifies how much the mean estimate itself varies from sample to sample.
4. A researcher observes that the invited group scored higher and subsequently chooses a one-tailed test (\(p = 0.04\) vs \(p = 0.08\)). What is methodologically wrong?
The test directionality was selected post hoc after inspecting the data. One-tailed tests are valid only when preregistered prior to data collection and when an effect in the reverse direction is practically impossible.
5. With \(N = 10,000\) students, a difference of 0.5 points on a 100-point wellbeing scale is highly significant (\(p < .001\)). Is it an important finding?
Almost certainly not. With massive sample sizes, trivial differences reach statistical significance. Scientific importance is evaluated through effect size (here \(d \approx 0.04\)) and clinical relevance, not \(p\)-values.
Do it yourself
These tasks have no starter code and no answers. Write your own code below as you would for a thesis. The datasets students, semesters, questionnaire, and first_sem are loaded.
1. Part-time students may report greater stress than full-time students. Compute stress scores from the questionnaire (reversing stress_4), state the hypotheses, compare groups with Welch’s \(t\)-test and Mann-Whitney \(U\) test, calculate Cohen’s \(d\), and write a reporting sentence.
2. Test whether first-semester study hours exceed 25 hours per week on average. State the hypotheses, run a one-sample \(t\)-test, and report results with a 95% confidence interval.
3. Test whether faculty affiliation is related to considering dropout using a chi-square test of independence. Check expected counts and compute Cramér’s \(V\).
Work on your own computer
The project contains the datasets and all four parts of this page as a Python script: exercises with blanks, conceptual review with answers in solutions.py, and open tasks.
Download chapter07.zip, unzip it, and open the folder in VS Code or your preferred Python IDE.
Prerequisites:
pip install pandas numpy matplotlib seaborn scipy statsmodels data2thesis