Playground: Chapter 5

From Research Question to Data

This page practises the ideas of Chapter 5 in four parts. Practise the chapter works through the book’s exercises, with starter code, hints, and solutions. Go further goes beyond the book, with new questions and statistical simulations. Check your understanding asks short conceptual questions whose answers open with a click. Do it yourself gives open tasks with no answers: you plan the research design yourself, as you will for your own thesis.

Much of this chapter is about thinking before coding, so several exercises ask for a written answer. Write it first, on paper or as a comment (a line starting with #) in a code space, then open the model answer. In the code exercises, replace each ______ with your own code and press Run Code.

Practise the chapter

These are the exercises at the end of Chapter 5, with the same numbers.

Exercise 1: Better research questions

Improve these research questions so that data could answer them: (a) “Is exercise good for students?” (b) “What makes supervisors effective?” For each, say whether your version is descriptive, relational, or causal.

  1. “Do graduate students who report regular weekly aerobic exercise achieve higher semester wellbeing scores than sedentary students?” is relational. “Does random assignment to a structured 8-week physical activity programme improve graduate student psychological wellbeing?” is causal, and needs random assignment to answer. (b) “Is the frequency of monthly supervisor advisory meetings associated with graduate students’ self-reported research progress satisfaction?” is relational. Each version names who is studied and what is measured; the vague words “good” and “effective” are replaced by measurable operational constructs.

Exercise 2: A falsifiable hypothesis

Write a falsifiable hypothesis, its null hypothesis, and the result that would count against it, for the question: “Do students with children study fewer hours per week?” Then run the comparison of first-semester study hours below.

NoteHint

Use "count" inside .agg() to count the number of students per category. A falsifiable hypothesis states in advance the direction of difference and the exact empirical result that would refute it.

TipSolution 
# H1: Students with children study fewer hours per week than students without children.
# H0: The two groups study the same number of hours on average.
# Against H1: Students with children reporting equal or higher mean study hours.
study_by_children = (
    first_sem.groupby("has_children")["study_hours"]
    .agg(students="count", mean_study_hours="mean")
)
print(study_by_children.round(2))

Both groups study approximately 27.6 hours a week (153 students with children, 447 without), so this empirical data provides zero support for the hypothesis. Because the hypothesis and the refutation criteria were pre-specified, this is a genuine scientific finding, not an analytical failure: the study asked an empirical question and observed no difference where common intuition anticipated one.

Exercise 3: Levels of measurement

For each variable in students, give its level of measurement. Then convert financial_worry into an ordered categorical with descriptive labels, and make a frequency table of it.

NoteHint

Pass labels as categories and set ordered=True.

TipSolution 
labels = ["Not at all", "A little", "Moderately", "Very", "Extremely"]
students["financial_worry_cat"] = pd.Categorical.from_codes(
    codes=students["financial_worry"].fillna(0).astype(int) - 1,
    categories=labels,
    ordered=True
)
print(students["financial_worry_cat"].value_counts(dropna=False).sort_index())

Levels of measurement in students: - Nominal: student_id, supervisor_id, gender, faculty, programme, study_mode, has_children, lives_away, workshop, considering_dropout. - Ordinal: employment, financial_worry. - Ratio: age, workshop_sessions.

Exercise 4: The null world with small groups

Change the chapter’s null-world simulation to small groups of 30 students each (total 60) instead of 150. Between which values do 95% of the chance differences now lie? What does this mean for a small study of the workshop?

NoteHint

Two groups of 30 students require np.repeat(..., 30) and size=60.

TipSolution 
np.random.seed(2026)
n_sims = 5000
chance_diffs_small = np.empty(n_sims)
small_groups = np.repeat(["Invited", "Not invited"], 30)

for i in range(n_sims):
    wellbeing = np.random.normal(loc=60, scale=11, size=60)
    shuffled = np.random.permutation(small_groups)
    diff = (
        wellbeing[shuffled == "Invited"].mean()
        - wellbeing[shuffled == "Not invited"].mean()
    )
    chance_diffs_small[i] = diff

q_lower, q_upper = np.percentile(chance_diffs_small, [2.5, 97.5])
print(f"95% bounds for n=30 per group: [{q_lower:.2f}, {q_upper:.2f}]")

With \(n=30\) per group, 95% of the chance differences span roughly −5.5 to +5.5 points—more than double the interval observed with \(n=150\) per group. A real 5-point intervention effect would fall squarely within the band expected by chance alone, meaning an underpowered study could not distinguish a genuine effect from random luck.

Exercise 5: Did the randomisation work?

Check the baseline equivalence of the random workshop invitation across three other baseline variables: study_mode, has_children, and lives_away.

NoteHint

Compute percentage for "Yes" responses: (x == "Yes").mean() * 100.

TipSolution 
balance_check = students.groupby("workshop").agg(
    part_time=("study_mode", lambda x: 100 * (x == "Part-time").mean()),
    children=("has_children", lambda x: 100 * (x == "Yes").mean()),
    lives_away=("lives_away", lambda x: 100 * (x == "Yes").mean())
).round(1)
print(balance_check)

The groups are well balanced at baseline: 28% vs 31% part-time, 23% vs 28% with children. The minor difference in living away from family (39% vs 46%) represents expected random fluctuation; random assignment balances groups on average, not identically.

Exercise 6: How many students?

Use Python’s statsmodels.stats.power.TTestIndPower to find how many students per group a study needs to detect a difference of 0.3 standard deviations with 80% power, and with 90% power (\(\alpha = 0.05\), two-sided).

NoteHint

Pass effect_size=0.30. Power values are 0.80 and 0.90.

TipSolution 
power_analyzer = TTestIndPower()
n_80 = power_analyzer.solve_power(effect_size=0.30, power=0.80, alpha=0.05, alternative="two-sided")
n_90 = power_analyzer.solve_power(effect_size=0.30, power=0.90, alpha=0.05, alternative="two-sided")

print(f"Required per group for 80% power: {int(np.ceil(n_80))}")
print(f"Required per group for 90% power: {int(np.ceil(n_90))}")

The calculation yields \(n \approx 176\) per group for 80% power and \(n \approx 235\) for 90% power (always round up). Elevating power from 80% to 90% requires one-third more participants.

Go further

These exercises go beyond the book.

Exercise 7: Kinds of questions

Classify each question as descriptive, relational, or causal, then open the model answer.

  1. What share of graduate students work in a paid job?
  2. Is financial worry associated with considering dropping out?
  3. Does a random invitation to a mindfulness course reduce stress?
  4. How does average sleep change over four semesters?
  5. Do students who meet their supervisor more often have higher GPAs?
  1. Descriptive: estimates a single univariate population proportion.
  2. Relational: tests an association between two observed variables.
  3. Causal: tests an intervention effect made possible by random assignment.
  4. Descriptive / Longitudinal: tracks temporal trajectory over time without attributing causality.
  5. Relational: observational association. Meeting frequency was self-selected, not experimentally randomised.

Exercise 8: Hypotheses that could be wrong

Rewrite each vague statement below into a falsifiable directional hypothesis that risks empirical refutation:

  1. “Stress affects students in various ways.”
  2. “The workshop works for students who engage with it properly.”
  3. “Social media use may be related to wellbeing.”
  1. “Students with higher baseline academic stress report lower semester GPAs”; refuted by a zero or positive correlation.
  2. “Students who attend at least four of the six sessions have higher semester 2 wellbeing than uninvited students”; refuted by equal or lower wellbeing.
  3. “Daily hours spent on social media correlate negatively with student sleep duration”; refuted by zero or positive correlation.

Exercise 9: Stratified vs simple random samples

Treat the 600 students as a population, and draw 2,000 samples of size 50 using simple random sampling vs stratified sampling (35 full-time, 15 part-time). Compare stability:

NoteHint

Select 15 part-time students to complete the sample of 50.

TipSolution 
for _ in range(2000):
    s_simple = first_sem.sample(n=50, replace=False)
    simple_means.append(s_simple["wellbeing"].mean())

    s_strat = pd.concat([
        full_time.sample(n=35, replace=False),
        part_time.sample(n=15, replace=False)
    ])
    stratified_means.append(s_strat["wellbeing"].mean())

print(f"Simple sample SD of estimates:     {np.std(simple_means):.3f}")
print(f"Stratified sample SD of estimates: {np.std(stratified_means):.3f}")

Simple random sampling yields variable part-time representation (5 to 25 students), whereas stratified sampling guarantees exactly 15. Because study mode explains little of the total variance in wellbeing, the standard deviations of the estimates are nearly identical (~1.62 vs ~1.63). Stratification primarily safeguards subgroup representation.

Exercise 10: Power for a correlation via Monte Carlo simulation

Estimate the power to detect a true population correlation \(r = 0.20\) at \(\alpha = 0.05\) across sample sizes \(n \in [100, 150, 200, 250]\):

NoteHint

The two-sided significance threshold is 0.05.

TipSolution 
def simulate_correlation_power(n, r=0.20, n_sims=1000):
    significant_count = 0
    for _ in range(n_sims):
        x = np.random.normal(size=n)
        y = r * x + np.sqrt(1 - r**2) * np.random.normal(size=n)
        r_val, p_val = stats.pearsonr(x, y)
        if p_val < 0.05:
            significant_count += 1
    return significant_count / n_sims

for sample_n in [100, 150, 200, 250]:
    power_est = simulate_correlation_power(sample_n, r=0.20)
    print(f"Sample size {sample_n}: Estimated power = {power_est:.2f}")

Estimated power is ~0.51 at \(n=100\), ~0.71 at \(n=150\), ~0.83 at \(n=200\), and ~0.87 at \(n=250\). Detecting a typical correlation of 0.20 with 80% power requires roughly 190 to 200 participants.

Exercise 11: A larger biased sample remains biased

Repeat the voluntary survey simulation where higher-wellbeing students are more likely to respond, using a sample of 400 out of 600 students. Does a huge sample eliminate bias?

NoteHint

Set size=400.

TipSolution 
voluntary_estimates_400 = [
    np.random.choice(pop_wellbeing, size=400, replace=True, p=p_answer).mean()
    for _ in range(2000)
]

print(f"True population mean:             {true_mean:.2f}")
print(f"Biased sample (n=400) mean:       {np.mean(voluntary_estimates_400):.2f}")
print(f"Standard error of estimate:       {np.std(voluntary_estimates_400):.3f}")

The true population mean is 60.5. Even with two-thirds of the population sampled (\(n=400\)), the voluntary estimates converge around 63.9 (over 3 points too high) with an extremely tight standard error (~0.30). A large biased sample merely delivers an erroneous estimate with elevated confidence!

Check your understanding

Answer each question in your own words first, then click to see a model answer.

1. Why must a hypothesis be stated before the data is analysed?

A hypothesis formed after observing the data merely redescribes empirical patterns (HARKing) and cannot fail. An a priori hypothesis ensures the empirical test genuinely risks refutation.

2. A statistical test rejects the null hypothesis. What has it NOT shown?

It has not shown that the research hypothesis is uniquely true, that the effect size is practically important, or that the association is causal (in an observational study where confounding remains). It demonstrates only that the observed data would be improbable under the assumption that nothing is occurring.

3. What is the difference between reliability and validity?

Reliability is measurement consistency/precision. Validity is whether the instrument measures the intended theoretical construct. A bathroom scale that consistently reads 2 kg too high is perfectly reliable, yet invalid.

4. Why is a larger biased sample still biased?

Sample size reduces random sampling variance (standard error), not systematic selection bias. If an online questionnaire primarily attracts motivated individuals, 10,000 responses describe motivated respondents more precisely, not the target population.

5. What does random assignment balance that measuring and adjusting for variables cannot?

Random assignment balances both measured AND unmeasured/unknown confounders across treatment arms on average. Statistical control can only adjust for variables that were explicitly anticipated, measured, and specified in the model.

Do it yourself

These tasks have no starter code and no answers. Write your own Python code, as you would for a thesis. The datasets (students, semesters, questionnaire, first_sem) and libraries (pandas, numpy, scipy, statsmodels) are preloaded.

1. For a topic from your own field, state: topic, problem, research question (descriptive, relational, or causal), falsifiable hypothesis (H1), null hypothesis (H0), and the empirical finding that would refute H1.

2. Create an analysis plan DataFrame for two hypotheses with columns: hypothesis, outcome, predictor, test, and decision_rule.

3. Check the baseline balance of the workshop invitation across all variables in students in a single unified table.

4. Load the toothgrowth.csv dataset (pd.read_csv("toothgrowth.csv")). Find the smallest standardized difference (\(d\)) that two groups of 10 animals can detect with 80% power at \(\alpha = 0.05\) using TTestIndPower. Write the resulting limitation.

Work on your own computer

NoteDownload the Chapter 5 project

The project contains the data and all four parts of this page as a standalone Python project: exercises with blanks (exercises.py), questions with model solutions (solutions.py), and open tasks.

  • Download chapter05.zip, unzip it, and open the folder in VS Code or your preferred Python IDE.

  • Make sure you have pandas, numpy, scipy, and statsmodels installed:

    pip install pandas numpy scipy statsmodels