By the end of today you can

• describe data as a rectangular table of cases and variables;
• identify the unit of analysis in your research design;
• create Pandas Series and Categoricals, and select and update values;
• select rows and columns from a DataFrame using .loc and .iloc;
• recognize 2D matrices and Python dictionaries in statistical output;
• import CSV, Excel, and SPSS files, keeping survey question labels;
• build an automated codebook and export clean data;
• diagnose and fix Python’s most common error tracebacks.

Data is a record of measurement

Each value records one observation about one case.

Before conducting any statistical test, clarify:

• what each number or text label stands for;
• what level of measurement generated it;
• how the values are structured in computer memory.

These answers determine which summaries, charts, and models are scientifically valid.

Cases and variables

Rows are cases, columns are variables. This data matrix is the foundation of nearly every quantitative method.

The unit of analysis

What one row represents is the unit of analysis, and a study often incorporates several.

The same student, four rows

  student_id  semester   gpa  sleep_hours  study_hours
0      S0001         1  3.02          7.2           26
1      S0001         2  3.27          8.4           17
2      S0001         3  3.17          7.8           25
3      S0001         4  3.00          7.2           28
4      S0002         1  3.25          6.4           32

A demographic question about students counts each student once (\(N = 600\)).

A question about change over time needs all 2,326 rows, and statistical models that recognize within-person repeated measures (Chapter 10).

Python data structures for research

flowchart LR
    S["Series<br>one variable"] --> C["Categorical<br>discrete groups"]
    S --> DF["DataFrame<br>cases × variables"]
    DF --> M["Matrix & Dict<br>statistical results"]

Pandas organizes data into intuitive scientific abstractions: Series for single measurement vectors, and DataFrames for rectangular tables.

A Pandas Series holds one variable

import pandas as pd

sleep = pd.Series([6.5, 7.0, 5.5, 8.0, 6.0])
print(sleep)

A Series is a 1-dimensional labeled vector holding values of one homogeneous type.

Type coercion when types are mixed

mixed = pd.Series([6.5, "seven", 5.5])
print(mixed)
# 0      6.5
# 1    seven
# 2      5.5
# dtype: object

Mixing numbers with text forces Pandas to assign object (string) dtype to the whole column.

The numbers become text strings and can no longer be averaged. A single stray word (e.g. "missing") in a survey export causes this.

Operations across a whole Series (Vectorization)

Operations apply to all values at once without loops:

# Convert hours to minutes in one step:
sleep * 60

# Identify nights shorter than 7 hours:
sleep < 7

Because True is 1 and False is 0 in numerical calculations:

print("Short nights count:", (sleep < 7).sum())    # 3
print("Short nights share:", (sleep < 7).mean())   # 0.60

0-based indexing and .iloc[]

Python uses 0-based indexing: the first element is at position 0:

print("First night:", sleep.iloc[0])           # 6.5
print("First and third:", sleep.iloc[[0, 2]])  # 6.5, 5.5

.iloc[] selects elements by integer position.

Boolean indexing (Logical filtering)

Pass a boolean condition inside square brackets:

sleep[sleep < 7]

Reads aloud: “give me sleep, where sleep is less than 7.”

Pandas extracts only the observations where the condition evaluates to True.

Updating values in place

Assigning to an indexed location updates the underlying data:

sleep.iloc[1] = 7.5
print(sleep)

The second observation is updated from 7.0 to 7.5 hours.

Managing qualitative groups: Categoricals

Qualitative categories (faculty, group, gender) are stored using Pandas category dtype:

faculty = pd.Series(
    ["Education", "Humanities", "Education", "Health Sciences"],
    dtype="category"
)
print(faculty.cat.categories)
# Index(['Education', 'Health Sciences', 'Humanities'], dtype='object')

Enforcing ordinal scale order

When categories possess an intrinsic hierarchy, specify ordered=True:

employment = pd.Series(
    pd.Categorical(
        ["None", "Full-time job", "Part-time job", "None"],
        categories=["None", "Part-time job", "Full-time job"],
        ordered=True
    )
)
print(employment.value_counts(sort=False))

This guarantees that tables and plots display categories from lowest to highest rather than alphabetically.

Levels of measurement in Python

The measurement scale dictates which statistical tests are meaningful (Chapter 5).

The DataFrame: A table of cases

pilot = pd.DataFrame({
    "student": ["S1", "S2", "S3", "S4"],
    "faculty": ["Education", "Humanities", "Education", "Health Sciences"],
    "sleep":   [6.5, 7.5, 5.5, 8.0],
    "invited": [True, False, True, False]
})

A DataFrame is a two-dimensional grid where each column is a Series.

Inspect its architecture with pilot.info().

Selecting columns and slicing rows

# Select one column as a Series:
pilot["sleep"]

# Select second row by position:
pilot.iloc[1, :]

# Select all rows for specific columns:
pilot.loc[:, ["student", "sleep"]]

# Filter rows meeting a condition:
pilot.loc[pilot["sleep"] < 7, :]

.iloc operates on integer offsets; .loc operates on labels and boolean conditions.

Filtering with multiple criteria

In Pandas, combine conditions using bitwise & (AND) and | (OR).

Crucial rule: wrap each individual condition in parentheses ():

hs_phd = students[
    (students["faculty"] == "Health Sciences") &
    (students["programme"] == "PhD")
]
print("Health Sciences PhD count:", len(hs_phd))

Because & has higher operator precedence than ==, omitting parentheses causes a ValueError.

Correlation matrices

Statistical procedures frequently return 2D numerical grids:

vars = first_sem[["gpa", "sleep_hours", "study_hours", "wellbeing"]]
correlations = vars.corr()
print(round(correlations, 2))

Extract specific bivariate associations using .loc[row, col]:

correlations.loc["sleep_hours", "wellbeing"]

Dictionaries and structured results

A dictionary (dict) maps keys to values:

student = {
    "id": "S0001",
    "programme": "Master's",
    "sleep": [7.2, 8.4, 7.8, 7.2]
}
print(student["sleep"])

Statistical packages like scipy.stats return results with named attributes:

import scipy.stats as stats
res = stats.ttest_1samp(first_sem["sleep_hours"], popmean=7.0, nan_policy="omit")
print(res.statistic, res.pvalue)

Importing research data formats

All reader functions return standard Pandas DataFrames ready for analysis.

Preserving survey question wording from SPSS

SPSS files embed the verbatim survey question in variable metadata:

import pyreadstat
spss_df, meta = pyreadstat.read_sav("wellbeing.sav")

print(meta.column_names_to_labels["stress_1"])
# "I feel unable to control important things in my studies."

Value labels are also preserved:

print(meta.variable_value_labels["faculty"])
# {1: 'Education', 2: 'Health Sciences', 3: 'Humanities', ...}

Building an automated codebook

codebook = pd.DataFrame({
    "variable": meta.column_names,
    "label":    meta.column_labels,
    "dtype":    [str(spss_df[c].dtype) for c in meta.column_names]
})
print(codebook.head(8))

An automated codebook documents every measured variable and provides an essential appendix for your thesis.

Exporting cleaned datasets

# Export to CSV (always set index=False):
students.to_csv("students_clean.csv", index=False)

# Export to Excel:
students.to_excel("students_clean.xlsx", index=False)

# Export to SPSS:
import pyreadstat
pyreadstat.write_sav(students, "students_clean.sav")

Never overwrite raw source data. Cleaned tables are saved under new names.

SPSS & Excel equivalents in Python

Diagnosing common Python tracebacks

Practical lab: Chapter 2 Playground

Test these skills interactively in your browser with Pyodide:

• Filter vectors with boolean masks
• Construct ordered categorical factors
• Subset multi-wave student datasets
• Download chapter02.zip to run directly in VS Code

Summary

A DataFrame is a rectangular matrix of cases and variables. Using Pandas Series, boolean indexing, and explicit categorical dtypes, every data manipulation step remains transparent, reproducible, and verifiable.

Next: Chapter 3

In Chapter 3, we address real-world data cleaning:

• Sanitizing raw survey exports (wellbeing_raw.xlsx);
• Diagnosing missing data mechanisms;
• Recoding survey values and reverse-coding scale items;
• Merging multi-wave tables across relational keys (student_id).