Playground: Chapter 17

Reproducible Research

This page practises the ideas of Chapter 17 in four parts. Practise the chapter works through the book’s exercises, with starter code, hints, and solutions. Go further goes beyond the book, with new questions and simulations. Check your understanding asks short questions whose answers open with a click. Do it yourself gives open tasks with no answers: you make your own work reproducible, as you will in your thesis.

Quarto documents and Shiny for Python apps run on your own computer, so the download project is the main practice for this chapter: a complete Quarto project with a results chapter, a parameterised faculty report, and the wellbeing dashboard, with tasks and finished versions. This page practises the parts that run in the browser: numbers that write themselves into sentences, protecting participants before sharing data, the garden of forking paths, and recording software versions. In the exercises, replace each ______ with your own code and press Run Code.

Practise the chapter

These are the exercises at the end of Chapter 17, with the same numbers.

Exercise 1: A sentence that writes itself

The chapter’s tiny Quarto document writes the average sleep of three friends into a sentence with inline code. Quarto runs on your own computer, but the idea can be tried here: build the sentence with a Python f-string, as inline code would, round the mean to one decimal place, then change one friend’s sleep and build it again.

NoteHint

In Python f-strings, format specifier :.1f formats a floating-point number to one decimal place.

TipSolution 
print(f"My three friends slept {np.mean(sleep):.1f} hours on average last night.")

The first sentence says 6.3 hours, and after the change, 6.7: the sentence followed the data without anyone retyping a number. Without rounding, it would say 6.333333333333333, which no reader needs. In the download project, the file results.qmd does the same in a real Quarto document; render it, change the data, and render again.

Exercise 2: A Quarto report of an earlier analysis

Turn one of your analyses from an earlier chapter into a Quarto document with a numbered table, a numbered figure, and a sentence with inline numbers. Render it to HTML and to Word.

This exercise needs Quarto, so it is done in the download project, whose results.qmd is a worked example to start from.

A table chunk gets a label starting with tbl- and a caption; a figure chunk a label starting with fig-; both are referred to in the text with @:

---
title: "Wellbeing by study mode"
format:
  html: default
  docx: default
jupyter: python3
---

```{python}
#| label: tbl-mode
#| tbl-cap: "First-semester wellbeing by study mode."
mode_summary = (
    first_sem.groupby("study_mode")["wellbeing"]
    .agg(Mean="mean", SD="std", Students="count")
    .reset_index()
    .round(1)
)
mode_summary
```

```{python}
#| label: fig-mode
#| fig-cap: "Wellbeing by study mode."
import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(6, 3.5))
sns.boxplot(data=first_sem, x="study_mode", y="wellbeing")
plt.tight_layout()
plt.show()
```

As @tbl-mode and @fig-mode show, part-time students' average wellbeing was lower.

Rendering produces an HTML file and a Word file from the same source. The finished solutions/results.qmd in the project shows a complete version.

Exercise 3: References and a citation style

Add two references to a references.bib file, cite them in the document, and switch the citation style with a csl: file.

This exercise needs Quarto, so it is done in the download project, which includes a references.bib.

The YAML header names the bibliography and, optionally, a style; citations use the keys in square brackets:

bibliography: references.bib
csl: apa.csl
Effect sizes were interpreted with Cohen's guidelines [@cohen1988], and
analyses were conducted in Python [@peng2011].

Quarto adds a reference list at the end. Citation styles are .csl files, freely available from the Zotero Style Repository for thousands of journals; downloading another one, such as vancouver.csl, and changing the csl: line changes every citation and the reference list at once, without touching the text.

Exercise 4: A slider in the dashboard

Add a slider to the Shiny for Python dashboard that chooses the range of semesters shown.

This exercise needs Shiny for Python, so it is done in the download project, whose app.py is the chapter’s dashboard.

The slider goes in the user interface, and the server filters the data by its value:

# in the UI sidebar:
ui.input_slider("semesters", "Semesters", min=1, max=4, value=(1, 4), step=1)

# in the server, in the reactive data:
@reactive.calc
def selected():
    s_range = input.semesters()
    df = wellbeing_data[wellbeing_data["faculty"] == input.faculty()]
    return df[(df["semester"] >= s_range[0]) & (df["semester"] <= s_range[1])]

A range slider returns a tuple of two numbers, the lower and upper bounds. Because the plot and the table both use selected(), they update together whenever the slider moves. The finished solutions/app.py in the project has the complete app.

Exercise 5: The smallest cells

Look at the smallest cells in the table of faculty, study mode, gender, and children. Decide which variable you would remove or group before sharing the data, and explain why.

NoteHint

The loop calculates smallest for each reduced table; print that variable inside the f-string.

TipSolution 
print(f"Removed: {drop:<14} Smallest cell: {smallest}")

Five of the 40 combinations describe fewer than 5 students; the smallest, such as part-time male students with children in Education, describe only 3. All the small cells are part-time students with children, a rare combination. Removing any one variable leaves no cell below 5: without faculty the smallest cell has 21 students, without children 11, without study mode 10, and without gender 7. The choice should depend on the research question: if the shared data is meant for studying students with children, keep that variable and remove, or group, another. Gender is often the least needed for the wellbeing questions and gives the smallest protection (7), so grouping faculties, which keeps every variable, may be better (Exercise 9).

Exercise 6: A sixth analysis

Add a sixth analysis to forking_paths(), a t-test that excludes the five highest scores, and describe how the rate of false positives for any analysis changes. (The simulation repeats 1,000 studies, so it takes a few seconds.)

NoteHint

Check whether any p-value in p_vals is less than 0.05.

TipSolution 
"any_analysis": any(p < 0.05 for p in p_vals)

The single planned analysis gives a false positive in about 5% of studies (~5.4%), but the chance that any of the six analyses does so rises to ~12.5% (compared to 10.9% with five analyses). Each extra defensible analysis adds another chance of a lucky result, although there is no real group difference.

Go further

These exercises go beyond the book.

Exercise 7: How many forking paths?

Extend the simulation to ten defensible analyses, and calculate the rate of false positives when a researcher tries the first 1, 2, 5, and 10 of them. Compare with what ten independent tests would give.

NoteHint

The theoretical rate if tests were independent is given by indep.

TipSolution 
print(f"Analyses tried: {k:<2} False positive rate: {any_sig:.3f} (If independent: {indep:.3f})")

Trying 1 analysis gives false positives in ~5.5% of studies, 2 in ~7.7%, 5 in ~10.9%, and 10 in ~14%. Ten independent tests would give 40% (1 − 0.95¹⁰). The analyses of one dataset are strongly related, since they all compare the same two groups on nearly the same scores, so each extra path adds less and less. Even so, a researcher who tries ten reasonable analyses and reports the best has almost three times the advertised risk of a false positive! Preregistration makes the planned path visible.

Exercise 8: A report for each programme

Change the chapter’s parameterised faculty report into a report for each programme (Master’s and PhD), and render it for both.

This exercise needs Quarto, so it is done in the download project, which includes faculty-report.qmd.

In the YAML header, the parameter changes from faculty to programme, and the code filters on it:

params:
  programme: "Master's"
prog_name = params['programme']
programme_semesters = merged[merged['programme'] == prog_name]

It is then rendered once for each value, from the Terminal:

quarto render programme-report.qmd -P programme:"Master's" -o report-masters.html
quarto render programme-report.qmd -P programme:"PhD" -o report-phd.html

One source file produces as many reports as there are programmes, all guaranteed to use the exact same analysis pipeline.

Exercise 9: Grouping instead of removing

Instead of removing a variable, group the five faculties into two broader groups, Sciences (Natural Sciences and Health Sciences) and Humanities and Social Sciences (the other three), and check that no cell below 5 remains.

NoteHint

Check for cells where n < 5.

TipSolution 
print("Small cells (< 5) remaining:", (grouped["n"] < 5).sum())

No cell below 5 remains: the smallest cell, part-time male students with children in the sciences, now has 8 students. Grouping keeps all four variables, so the shared data can still answer questions about gender, study mode, and children; it gives up detail on the one variable that least needs it.

Exercise 10: Recording the software

Record the Python version and the version of statsmodels used, and format them into a methodology sentence.

NoteHint

In standard Python packages, the version string is stored in the __version__ attribute.

TipSolution 
sm_ver = statsmodels.__version__

The sentence reports the versions actually used, dynamically constructed by code so that it is always accurate. In Python, pip freeze > requirements.txt records every package version so that the environment can be restored exactly.

Check your understanding

Answer each question in your own words first, then click to see a model answer.

1. What is the difference between a reproducible result and a replicable one?

A result is reproducible if the same data and the same code give the same result when someone else runs them. It is replicable if a new study, with new data collected in the same way, finds the same thing. Reproducibility checks the analysis; replication checks the finding. A result can be perfectly reproducible and still fail to replicate.

2. Why should numbers in the text of a thesis not be typed by hand?

Because typed numbers do not change when the data or the analysis changes. After a correction to the data, a hand-typed mean, p-value, or sample size silently goes out of date, and errors of copying creep in. Inline code calculates every number from the data when the document is rendered, so the text always matches the analysis.

3. What does requirements.txt / lockfile record, and why does it matter?

The exact version of every package a project uses. Packages change between versions, sometimes altering defaults or numerical routines; with a requirements or lock file, the project can be restored with the exact same versions on another computer or years later, ensuring the analysis reproduces cleanly.

4. Why is removing names and student numbers not enough to anonymise data?

Because combinations of ordinary variables can identify people. A part-time male student with children in one faculty may be one of only three people, and anyone who knows the department could recognise him. Small cells must be protected, by grouping categories, removing variables that are not needed, or sharing only summaries, and what may be shared is limited by what participants consented to.

5. What does preregistration protect against?

Against choosing the analysis after seeing the results (the garden of forking paths), in which trying several reasonable analyses and reporting the one that works raises the rate of false positives far above 5%. By recording the hypotheses and planned analyses in advance, preregistration separates confirmatory results from exploratory ones, so readers know which findings were predicted.

Do it yourself

These tasks have no starter code and no answers. The code space is empty and ready to run; the data (students and semesters) and standard packages are already imported.

1. Choose a result from an earlier chapter, such as the difference in wellbeing between part-time and full-time students, and write the reporting sentence as code: every number in it (the means, the difference, the test statistic, the p-value, and the sample sizes) calculated and formatted using Python f-strings, rounded as a journal would print them.

2. In the download project, turn one analysis from Chapters 6 to 10 into a Quarto report with a numbered table, a figure, inline numbers, and one citation, and render it to Word. Then add a second input to the Shiny for Python dashboard, such as study mode or programme, and explain in comments how the inputs, the reactive data, and the outputs are linked.

3. Write a preregistration for a study of your own, using the headings of the AsPredicted template: the main question and hypotheses, the dependent variable, the conditions or groups, the planned analyses, the rules for excluding data, the sample size and how it was chosen, and anything else to preregister.

Work on your own computer

NoteDownload the Chapter 17 project

A complete Quarto and Python project: results.qmd, a results chapter with inline numbers, a numbered table and figure, and citations; faculty-report.qmd, a report with a faculty parameter; and app.py, the wellbeing dashboard. exercises.txt lists the Quarto and Shiny tasks, and the solutions folder has finished versions. exercises.py and solutions.py hold the Python exercises of this page.