Playground: Chapter 6

Descriptive Statistics and Exploratory Data Analysis

This page practises the ideas of Chapter 6 in four parts. Practise the chapter works through the book’s exercises, with starter code, hints, and solutions. Go further goes beyond the book, with new questions and statistical simulations. Check your understanding asks short questions whose answers open with a click. Do it yourself gives open tasks with an empty code space and no answers: you decide how to describe the data, as you will in your own thesis.

In the exercises, replace each ______ with your own code and press Run Code.

Practise the chapter

These are the exercises at the end of Chapter 6, with the same numbers.

Exercise 1: Four summaries of study hours

Calculate the mean, median, SD, and IQR of first-semester study_hours. Decide whether the variable is closer to symmetrical or skewed, and which pair of summaries you would report.

Solution

sh = first_sem["study_hours"]
mean_sh   = sh.mean()
median_sh = sh.median()
sd_sh     = sh.std()
iqr_sh    = sh.quantile(0.75) - sh.quantile(0.25)

print(f"Mean:   {mean_sh:.2f}")
print(f"Median: {median_sh:.2f}")
print(f"SD:     {sd_sh:.2f}")
print(f"IQR:    {iqr_sh:.2f}")

The mean (27.6 hours) is slightly above the median (26.0 hours): study hours are mildly skewed to the right, with a tail of students who study very long hours. The skewness is small (0.65), so reporting the mean and SD (27.6 and 13.1) is acceptable; the median and IQR (26.0 and 19.0) would be the safer choice if the tail were longer. Whichever pair you choose, always report both centre and spread together.

Exercise 2: Is wellbeing normal?

Draw a Q-Q plot of first-semester wellbeing, and judge whether it looks normal.

Solution

fig, ax = plt.subplots(figsize=(6, 4))
sm.qqplot(first_sem["wellbeing"].dropna(), line="s", ax=ax)
ax.set_title("Q-Q Plot for First-Semester Wellbeing")
plt.show()

The sample quantiles follow the straight reference line closely across almost the entire range, with only minor deviations at the extreme tails. First-semester wellbeing is close enough to normally distributed for standard parametric tests.

Exercise 3: Unusually high study hours

Using Tukey’s box plot rule (1.5 × IQR), count the students with unusually high study_hours in the first semester, and inspect their other variables.

Solution

x = first_sem["study_hours"]
q1 = x.quantile(0.25)
q3 = x.quantile(0.75)
iqr = q3 - q1
fence = q3 + 1.5 * iqr

outliers = first_sem.query("study_hours > @fence")[
    ["student_id", "study_hours", "sleep_hours", "gpa", "wellbeing", "study_mode"]
]
print(f"Upper fence: {fence:.1f} hours")
print(f"Number of outliers: {len(outliers)}")
print(outliers.head())

The upper fence is around 64.5 hours per week. A few students report studying 65 to 75 hours per week. While extreme, these numbers are plausible for dedicated full-time graduate students with approaching dissertation deadlines. They should be retained in the analysis rather than pruned.

Exercise 4: Sleep of students who left

Compare the first-semester sleep_hours of students who later left the study with those who stayed, and say whether they differ in sleep as they did in wellbeing.

Solution

sem3_students = set(semesters.query("semester == 3")["student_id"])
left_ids = set(students["student_id"]) - sem3_students

first_sem["left_study"] = first_sem["student_id"].isin(left_ids)
summary = first_sem.groupby("left_study").agg(
    mean_sleep=("sleep_hours", "mean"),
    mean_wellbeing=("wellbeing", "mean"),
    students=("student_id", "count")
).round(2)
print(summary)

The students who left slept an average of 6.13 hours in semester 1, compared to 6.44 hours for those who stayed—a modest sleep deficit of about 19 minutes. However, their wellbeing deficit was severe (45.3 vs 53.6). Dropouts experienced broader emotional distress and burnout rather than isolated sleep loss.

Exercise 5: Support and satisfaction

Find the correlation between supervisor support and academic satisfaction, classify it with Cohen’s guidelines (0.1 small, 0.3 medium, 0.5 large), and explain why it does not show that support causes satisfaction.

Solution

scores = pd.DataFrame({
    "student_id": questionnaire["student_id"],
    "support": questionnaire.filter(like="support_").mean(axis=1),
    "satisfaction": questionnaire.filter(like="satisfaction_").mean(axis=1),
})
r_value = scores["support"].corr(scores["satisfaction"])
print(f"Pearson r: {r_value:.2f}")

The correlation is approximately \(r = 0.48\), which Cohen classifies as a medium-to-large positive correlation: students who feel better supported by their supervisors report higher satisfaction with their degree programme. However, correlation is not causation. A third variable (such as department funding, lab culture, or overall student morale) might boost both support and satisfaction.

Exercise 6: A summary for every variable

For each variable in students, choose the summary you would report in a thesis, using the chapter’s table of summaries by level of measurement and shape, and justify each choice in one sentence. Write your answer first, then open the model answer.

Model answer

age: Continuous ratio variable with right skew (older returning students) \(\rightarrow\) report median and IQR (or mean and SD with a note on skew).
gender, nationality, faculty, programme, study_mode, has_children, lives_away, workshop, considering_dropout: Nominal variables \(\rightarrow\) report counts and percentages in each category (and the mode).
employment and financial_worry: Ordinal variables \(\rightarrow\) report percentages in each category and the median; intervals between steps are not strictly equal, so computing a mean implies false precision.
workshop_sessions: Discrete count variable \(\rightarrow\) report median and IQR (or the percentage of students attending each session count).
student_id and supervisor_id: Unique identifiers \(\rightarrow\) do not summarise, though the distribution of students per supervisor is informative to report.

Go further

These exercises go beyond the book.

Exercise 7: The standard deviation by hand

Five students sleep 4, 6, 7, 8, and 10 hours. Build their standard deviation step by step: deviations from the mean, squared deviations, sample variance, and the square root. Then compare with Pandas .std().

Solution

sleep_five = pd.Series([4.0, 6.0, 7.0, 8.0, 10.0])
deviations = sleep_five - sleep_five.mean()
print("Deviations:", deviations.values)

squared_devs = deviations ** 2
variance = squared_devs.sum() / (len(sleep_five) - 1)
sd_manual = np.sqrt(variance)
sd_pandas = sleep_five.std()

print(f"Manual SD: {sd_manual:.4f}")
print(f"Pandas SD: {sd_pandas:.4f}")

The deviations are −3, −1, 0, 1, and 3; they sum to exactly 0, which is why we square them before averaging. The squared deviations sum to 20; dividing by \(5 - 1 = 4\) gives a sample variance of 5.0, and \(\sqrt{5} \approx 2.24\) hours, which matches sleep_five.std() perfectly.

Exercise 8: Mean and median of age

Compare the mean, median, and skewness of student age, and visualise the distribution with a histogram.

Solution

mean_age   = students["age"].mean()
median_age = students["age"].median()
skew_age   = students["age"].skew()

print(f"Age Mean: {mean_age:.1f}, Median: {median_age:.1f}, Skewness: {skew_age:.2f}")

fig, ax = plt.subplots(figsize=(6, 3.5))
sns.histplot(students["age"].dropna(), bins=20, color="#1d588c", ax=ax)
ax.axvline(mean_age, color="red", linestyle="-", label="Mean")
ax.axvline(median_age, color="orange", linestyle="--", label="Median")
ax.set_xlabel("Age (years)")
ax.legend()
plt.show()

The mean age is 29.7 years while the median is 29.0 years. The positive skewness (1.18) reflects the tail of older students stretching up to 52 years, pulling the mean upward while the median stays anchored at the central participant.

Exercise 9: Where are the gaps?

Count missing values in every column of semesters, grouped by semester.

Solution

missing_by_sem = semesters.groupby("semester").apply(
    lambda df: df.isna().sum()
)
print(missing_by_sem)

Missing values are sparse (at most 7-8 entries per semester, confined to self-reported sleep, exercise, study hours, and caffeine). GPA and wellbeing have zero missing records among active students. The major gap—the 52 students who dropped out prior to semester 3—is not visible as NaN cells, because entire rows are absent.

Exercise 10: When one mean misleads (bimodal distribution)

Using the Old Faithful faithful dataset, calculate the overall mean waiting time, then split the data into short and long waits using a cutoff of 67 minutes.

Solution

mean_wait = faithful["waiting"].mean()
print(f"Overall mean waiting time: {mean_wait:.1f} minutes")

faithful_grouped = faithful.copy()
faithful_grouped["wait_type"] = np.where(faithful_grouped["waiting"] < 67, "short", "long")
print(faithful_grouped.groupby("wait_type")["waiting"].agg(["count", "mean", "std"]))

The overall mean waiting time is 70.9 minutes. However, the distribution is strongly bimodal, with peaks near 54 minutes and 80 minutes. The mean sits in the low-frequency valley between the two peaks: very few eruptions occur after a 70-minute wait! Splitting by wait_type reveals two distinct modes: short waits (\(M = 54.5\) min, \(SD = 5.9\)) and long waits (\(M = 80.0\) min, \(SD = 5.9\)).

Check your understanding

Answer each question in your own words first, then click to see a model answer.

1. Which summary suits an ordinal variable, such as the five answers to a question about money worries?

Answer

The percentage in each category, and the median as the centre. An ordinal variable has a meaningful order, so the middle answer makes sense, but its intervals are not necessarily equal. Computing a mean (which assumes equal distances between “a little” and “somewhat”) suggests a mathematical precision the measurement does not possess.

2. A report says that sleep has a mean of 6.5 hours and an SD of 1.1 hours. What does the SD mean in words?

Answer

A typical student’s sleep duration is approximately 1.1 hours away from the sample mean of 6.5 hours. If sleep is normally distributed, roughly 68% of students sleep between 5.4 and 7.6 hours per night, and about 95% sleep between 4.3 and 8.7 hours.

3. Why is an outlier not simply deleted?

Answer

Because an unusual value is usually a legitimate observation from the real world. Deleting it artificially reduces natural variance, biases effect estimates, and misleads readers about the true variability of the phenomenon. Instead, verify that it is not a typographical error, report it transparently, and use robust statistics (median, IQR) or sensitivity analyses.

4. What is the difference between data missing at random (MAR) and data missing not at random (MNAR)?

Answer

Missing at random (MAR) means missingness depends systematically on observed variables that were recorded (e.g. part-time students missing meetings), so models controlling for those variables remain unbiased. Missing not at random (MNAR) means missingness depends on the unobserved value itself (e.g. students suffering severe financial distress refusing to disclose finances), creating inescapable bias among respondents.

5. Two variables have a correlation of 0. Does that mean they are unrelated?

Answer

No. Pearson’s correlation coefficient \(r\) detects only linear (straight-line) relationships. A strong non-linear or inverted-U relationship (such as performance peaking at moderate arousal and dropping at extremes) can produce a correlation of zero despite a strong functional dependency.

Do it yourself

These tasks have no starter code and no answers. Write your own code, as you would for a thesis. The datasets students, semesters, questionnaire, and first_sem are already loaded.

1. Produce a complete “Table 1” describing the sample: total student count, and for each baseline variable the percentages, or mean and SD, or median and IQR, with the number of missing values. Then write a short paragraph describing the sample, with every number dynamically calculated.

2. Describe a variable not covered in detail in the chapter: supervisor_meetings in the first semester. Compute its centre, spread, shape, and inspect any unusual values. Decide which summaries to report, and write two explanatory sentences.

3. Compare students sleeping \(< 6\) hours with those sleeping \(\ge 7\) hours on first-semester GPA and wellbeing. Report the mean, SD, and sample size for both groups.

Work on your own computer

Download the Chapter 6 project

The project contains the datasets and all four parts of this page as a Python script: exercises with blanks, conceptual questions with answers in solutions.py, and open tasks with room for your code.

Download chapter06.zip, unzip it, and open the folder in VS Code or your preferred Python IDE.

Prerequisites:

pip install pandas numpy matplotlib seaborn scipy statsmodels data2thesis