2 Data Structures in Python

Data is the recorded outcome of scientific measurement. Each value captures one observation about one entity: a student’s age, the faculty they belong to, the hours they slept during a semester, or their response to a Likert questionnaire item. Before conducting any statistical analysis, a researcher must clarify what each value signifies, what level of measurement generated it, and how the data is organized in computer memory. These foundations determine which summary statistics, visualizations, and inferential models are scientifically valid.

Research data rarely arrives in a single pristine file. In Elaf’s wellbeing study, the online survey software generated an Excel export (wellbeing_raw.xlsx), the university registrar exported semester records as CSV files (semesters.csv), and a collaborator from a previous pilot study stored historical records in SPSS (wellbeing.sav). Before analyzing these heterogeneous sources, we need to understand how Python organizes data in memory. Python provides specialized structures tailored for data science: lists and NumPy arrays for basic sequences, Pandas Series for single measurement variables, Categorical Series for discrete groups, and DataFrames for rectangular tables of cases. This chapter introduces these structures, demonstrates how to import files from every common format, and shows how to generate an automated codebook linking every column to its underlying survey question.

TipBy the end of this chapter you will be able to
  • Conceptualize data as a rectangular matrix of cases and variables, and identify the unit of analysis.
  • Create Pandas Series, perform vectorized arithmetic, and filter observations with boolean indexing.
  • Manage categorical and ordinal measurements using Pandas Categorical types.
  • Construct, inspect, and slice two-dimensional DataFrames using .loc and .iloc.
  • Understand dictionaries and 2D matrices when encountered in statistical output.
  • Import real-world data from CSV, Excel (.xlsx), and SPSS (.sav) files, preserving survey question labels.
  • Build an automated codebook that documents variable definitions and measurement scales.
  • Map familiar SPSS and Excel procedures to Python idioms, and debug common Python error tracebacks.

2.1 Data as measurement

Almost all empirical research data can be conceptualized as a rectangular grid: each row represents a case (or observation), and each column represents a variable. A case is the individual unit under observation—a student, a patient, an experimental plot, or an institution. A variable is a measured property recorded across all cases, such as chronological age, anxiety score, or study hours. Every cell in the grid contains a single value representing the measurement of that variable for that specific case. This structure, known as the data matrix, forms the backbone of statistical modeling.

What constitutes a case depends on the research design, defining the unit of analysis. Elaf’s wellbeing study involves multiple units of analysis across different tables. In the students table, each row represents an individual student (\(N = 600\)). In the semesters table, however, each row represents a single student in a specific academic semester, meaning each student appears up to four times:

semesters.head(5)
student_id semester gpa sleep_hours study_hours exercise_days caffeine_mg supervisor_meetings wellbeing
0 S0001 1 3.02 7.2 26.0 5.0 130.0 0 74
1 S0001 2 3.27 8.4 17.0 3.0 80.0 1 69
2 S0001 3 3.17 7.8 25.0 7.0 105.0 3 71
3 S0001 4 3.00 7.2 28.0 4.0 145.0 2 74
4 S0002 1 3.25 6.4 32.0 2.0 210.0 3 60

The first four rows all correspond to student S0001, documenting their trajectory across semesters 1 through 4. Confusing the unit of analysis is one of the most frequent errors in quantitative research. A demographic question concerning baseline enrollment must count each student exactly once. Conversely, an inquiry into GPA trajectories across time requires repeated-measures observations, coupled with statistical techniques that account for within-subject dependency (?@sec-ch10). In ?@sec-ch05, we explore how research hypotheses dictate the appropriate unit of analysis.

2.2 Variables and their values

In Python’s data science ecosystem, a single variable’s values are held in a Pandas Series (or a NumPy 1D array). A Series is an indexed, one-dimensional array capable of holding numbers, text, or boolean flags:

sleep = pd.Series([6.5, 7.0, 5.5, 8.0, 6.0])
sleep
0    6.5
1    7.0
2    5.5
3    8.0
4    6.0
dtype: float64

A Series enforces a single homogeneous data type (dtype). If you inadvertently mix numbers with text, Pandas converts the entire Series to an object (or string) dtype to accommodate the text:

mixed = pd.Series([6.5, "seven", 5.5])
mixed
0      6.5
1    seven
2      5.5
dtype: object

Notice that the numbers 6.5 and 5.5 have become text strings. They can no longer be averaged or plotted numerically. This behavior is vital to understand: if a single survey respondent types "seven" or "N/A" into a numeric field in a web form, the entire column imports into Python as text. We will encounter this exact issue when inspecting the raw Excel export later in this chapter.

2.2.1 Operations on a whole Series (Vectorization)

A defining strength of Pandas is vectorization: mathematical and logical operations apply across every value in the collection simultaneously, without needing explicit for loops. Converting sleep durations from hours to minutes requires a single multiplication:

sleep * 60
0    390.0
1    420.0
2    330.0
3    480.0
4    360.0
dtype: float64

Logical comparisons work identically, generating a boolean Series containing True or False for each entry. For example, identifying nights falling short of the clinical recommendation of 7 hours:

sleep < 7
0     True
1    False
2     True
3    False
4     True
dtype: bool

Because Python evaluates True as 1 and False as 0 in numerical contexts, calling .sum() counts the qualifying nights, and .mean() calculates their proportion:

print("Count of short nights:", (sleep < 7).sum())
print("Proportion of short nights:", (sleep < 7).mean())
Count of short nights: 3
Proportion of short nights: 0.6

Three of the five nights (60%) were short. This vectorized code operates at identical speed whether analyzing 5 observations or 500,000 observations.

2.2.2 Selecting values

Python uses 0-based indexing: the first element resides at position 0, the second at position 1, and so on. Think of an index as the offset or distance from the start of the array.

To select values by positional integer index in a Pandas Series, use .iloc[]:

print("First night:", sleep.iloc[0])
print("First and third nights:\n", sleep.iloc[[0, 2]])
First night: 6.5
First and third nights:
 0    6.5
2    5.5
dtype: float64

Values can also be extracted using a boolean condition inside square brackets. Pandas filters the Series, retaining only observations where the condition evaluates to True:

sleep[sleep < 7]
0    6.5
2    5.5
4    6.0
dtype: float64

This syntax, termed boolean indexing (or logical filtering), reads naturally: “give me the elements of sleep where sleep is less than 7.” It is one of the most widely used idioms in quantitative Python.

2.2.3 Changing values

Placing an indexed selection on the left side of the assignment operator updates the underlying data. If a participant later clarifies that their second night was 7.5 hours rather than 7.0:

sleep.iloc[1] = 7.5
sleep
0    6.5
1    7.5
2    5.5
3    8.0
4    6.0
dtype: float64

2.3 Categories (Categorical Data)

Research datasets frequently contain qualitative categories: faculty division, experimental arm, gender identity, or Likert response levels. Pandas represents categorical variables using the specialized category dtype (similar to categorical variables in statistics or value labels in SPSS). A categorical Series displays text labels but stores them internally as compact integers mapped to a defined list of categories:

faculty = pd.Series(
    ["Education", "Humanities", "Education", "Health Sciences"],
    dtype="category"
)
print(faculty)
print("Categories:", faculty.cat.categories)
0          Education
1         Humanities
2          Education
3    Health Sciences
dtype: category
Categories (3, str): ['Education', 'Health Sciences', 'Humanities']
Categories: Index(['Education', 'Health Sciences', 'Humanities'], dtype='str')

By default, categories are arranged in alphanumeric order. However, when categories possess an intrinsic hierarchy (such as employment intensity or Likert scales), you can define an ordered category using pd.Categorical:

employment = pd.Series(
    pd.Categorical(
        ["None", "Full-time job", "Part-time job", "None"],
        categories=["None", "Part-time job", "Full-time job"],
        ordered=True
    )
)
employment.value_counts(sort=False)
None             2
Part-time job    1
Full-time job    1
Name: count, dtype: int64

Specifying explicit category orders ensures that subsequent summary tables and plots display categories logically from lowest to highest rather than alphabetically.

2.3.1 Levels of measurement and Python types

The distinction between continuous numbers and categories is not merely aesthetic; it reflects the underlying mathematical properties of the measurement. Categories without an order (such as faculty or academic discipline) are nominal. Categories with a strict ranking but unequal or unquantified intervals (such as Likert scales or employment tiers) are ordinal. Measurements with uniform, quantifiable units (such as sleep hours, age, or GPA) are interval or ratio (numeric). Table 2.1 outlines how each measurement level corresponds to a Python data type.

Table 2.1: Levels of measurement and their representation in Python
Measurement Level Example Variables Python Data Type
Nominal (unordered categories) Faculty, Gender, Department category (or object / string)
Ordinal (ordered categories) Agreement scale, Education level category (ordered=True)
Interval / Ratio (continuous quantities) Sleep hours, Age, GPA, Scale sum float64, int64
Binary / Dichotomous Invited to workshop (Yes/No) bool or two-level category

2.4 A table of cases: The DataFrame

Most empirical data is organized as a two-dimensional table where each row represents a case and each column represents a variable. In Python, this table is a Pandas DataFrame. Each column is a Series sharing a common index. While each column must have a uniform data type, different columns across the DataFrame can store different types.

You can construct a DataFrame manually from a Python dictionary:

pilot = pd.DataFrame({
    "student": ["S1", "S2", "S3", "S4"],
    "faculty": pd.Categorical(["Education", "Humanities", "Education", "Health Sciences"]),
    "sleep": [6.5, 7.5, 5.5, 8.0],
    "invited": [True, False, True, False]
})
pilot
student faculty sleep invited
0 S1 Education 6.5 True
1 S2 Humanities 7.5 False
2 S3 Education 5.5 True
3 S4 Health Sciences 8.0 False

The most comprehensive way to inspect a DataFrame’s architecture is the .info() method, which summarizes column names, non-null counts, and data types:

pilot.info()
<class 'pandas.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 4 columns):
 #   Column   Non-Null Count  Dtype   
---  ------   --------------  -----   
 0   student  4 non-null      str     
 1   faculty  4 non-null      category
 2   sleep    4 non-null      float64 
 3   invited  4 non-null      bool    
dtypes: bool(1), category(1), float64(1), str(1)
memory usage: 228.0 bytes

2.4.1 Selecting columns and rows

To extract a single column as a Series, use square bracket notation with the column name:

pilot["sleep"]
0    6.5
1    7.5
2    5.5
3    8.0
Name: sleep, dtype: float64

To slice rows and columns simultaneously, Pandas provides two primary indexers: 1. .iloc[row_pos, col_pos]: selects by integer position (0-based offset). 2. .loc[row_label, col_label]: selects by label name or boolean mask.

# Select the second row across all columns
pilot.iloc[1, :]

# Select all rows for two specific columns
pilot.loc[:, ["student", "sleep"]]

# Filter rows where sleep is under 7 hours
pilot.loc[pilot["sleep"] < 7, :]
student faculty sleep invited
0 S1 Education 6.5 True
2 S3 Education 5.5 True

The final expression combines a DataFrame with boolean indexing: “retrieve all columns for rows where sleep is less than 7.” This is Python’s direct equivalent of SPSS’s Select Cases or Excel’s Filter.

2.4.2 Adding new columns

Assigning an expression to a new column name appends it to the DataFrame:

pilot["short_sleep"] = pilot["sleep"] < 7
pilot
student faculty sleep invited short_sleep
0 S1 Education 6.5 True True
1 S2 Humanities 7.5 False False
2 S3 Education 5.5 True True
3 S4 Health Sciences 8.0 False False

2.4.3 The student wellbeing cohort

The students dataset bundled in data2thesis follows this exact DataFrame architecture across 600 participants:

students.info()
<class 'pandas.DataFrame'>
RangeIndex: 600 entries, 0 to 599
Data columns (total 14 columns):
 #   Column               Non-Null Count  Dtype  
---  ------               --------------  -----  
 0   student_id           600 non-null    str    
 1   supervisor_id        600 non-null    str    
 2   age                  599 non-null    float64
 3   gender               600 non-null    str    
 4   faculty              600 non-null    str    
 5   programme            600 non-null    str    
 6   study_mode           600 non-null    str    
 7   employment           600 non-null    str    
 8   has_children         600 non-null    str    
 9   lives_away           600 non-null    str    
 10  financial_worry      562 non-null    float64
 11  workshop             600 non-null    str    
 12  workshop_sessions    600 non-null    int64  
 13  considering_dropout  600 non-null    str    
dtypes: float64(2), int64(1), str(11)
memory usage: 65.8 KB

In this output, object indicates text strings, int64 denotes integers, and float64 denotes decimal numbers. Columns representing categories, such as faculty, can be converted to categorical types whenever discrete factor levels are needed:

students["faculty"] = students["faculty"].astype("category")
students["faculty"].cat.categories
Index(['Education', 'Health Sciences', 'Humanities', 'Natural Sciences',
       'Social Sciences'],
      dtype='str')

Subsetting with multiple criteria uses Python’s bitwise logical operators: & (AND) and | (OR). In Pandas, each individual condition must be enclosed in parentheses () because bitwise operators take precedence over comparison operators (==, >, <):

hs_phd = students[
    (students["faculty"] == "Health Sciences") & (students["programme"] == "PhD")
]
print("Health Sciences PhD candidates:", len(hs_phd))
Health Sciences PhD candidates: 44
TipInspecting Data Visually in VS Code

In VS Code, opening any DataFrame in the Interactive Window or Data Viewer provides an interactive spreadsheet grid where you can sort, filter, and inspect columns visually without altering the underlying code.

2.5 Matrices and Dictionaries

Beyond Series and DataFrames, two additional data structures frequently appear in quantitative workflows: 2D numerical matrices and dictionaries.

2.5.1 Matrices and correlation grids

A numerical matrix is a two-dimensional grid of homogeneous numbers. Matrices arise naturally as the mathematical output of statistical procedures, such as calculating bivariate correlation coefficients across multiple metrics:

first_semester = semesters[semesters["semester"] == 1]
metric_vars = first_semester[["gpa", "sleep_hours", "study_hours", "wellbeing"]]

# Compute bivariate correlation matrix
correlations = metric_vars.corr()
round(correlations, 2)
gpa sleep_hours study_hours wellbeing
gpa 1.00 0.25 0.06 0.32
sleep_hours 0.25 1.00 -0.56 0.46
study_hours 0.06 -0.56 1.00 -0.24
wellbeing 0.32 0.46 -0.24 1.00

Each variable correlates perfectly with itself (\(r = 1.00\)) along the main diagonal. Specific pairwise correlations can be extracted using .loc[row, col]:

correlations.loc["sleep_hours", "wellbeing"]
np.float64(0.45630448626446257)

2.5.2 Dictionaries and statistical results

A dictionary (dict) is Python’s fundamental key-value container. It stores named attributes enclosed in curly braces {}:

student = {
    "id": "S0001",
    "programme": "Master's",
    "sleep": [7.2, 8.4, 7.8, 7.2]
}
student["sleep"]
[7.2, 8.4, 7.8, 7.2]

While researchers rarely build complex dictionaries by hand, many Python scientific packages return statistical test results as structured objects or dictionary-like containers. For instance, running a one-sample \(t\)-test with scipy.stats to test whether first-semester students sleep 7 hours on average:

import scipy.stats as stats

result = stats.ttest_1samp(first_semester["sleep_hours"], popmean=7.0, nan_policy="omit")
print("t-statistic:", round(result.statistic, 2))
print("p-value:", result.pvalue)
t-statistic: -12.18
p-value: 1.331700973081752e-30

The resulting object encapsulates the computed test statistic, degrees of freedom, and \(p\)-value as named attributes ready for automated reporting in your thesis.

2.6 Importing research data

Different software packages store tabular data in distinct binary or text formats. Python provides dedicated reader functions for every standard format, each parsing the input directly into a clean Pandas DataFrame. The sample files below are installed with data2thesis and located via dt.example_file():

2.6.1 CSV files

A CSV file (comma-separated values) is a plain-text table where commas delimit adjacent columns. Because CSV files are universally supported and human-readable, they represent the gold standard for long-term open science archiving. In Python, CSV files are imported using pd.read_csv():

path = dt.example_file("semesters.csv")
semester_data = pd.read_csv(path)
semester_data.head(3)
student_id semester gpa sleep_hours study_hours exercise_days caffeine_mg supervisor_meetings wellbeing
0 S0001 1 3.02 7.2 26.0 5.0 130.0 0 74
1 S0001 2 3.27 8.4 17.0 3.0 80.0 1 69
2 S0001 3 3.17 7.8 25.0 7.0 105.0 3 71

In your own thesis project, you will typically point directly to your project’s data directory: pd.read_csv("data/semesters.csv").

2.6.2 Excel spreadsheets

Real-world survey platforms frequently export multi-column workbooks (.xlsx). Pandas reads modern Excel files via pd.read_excel() (powered by the openpyxl engine):

raw_path = dt.example_file("wellbeing_raw.xlsx")
raw_data = pd.read_excel(raw_path)
print("Dimensions (rows, columns):", raw_data.shape)
Dimensions (rows, columns): (615, 64)

This file represents an unfiltered export from the wellbeing survey. The .shape property indicates r raw_data.shape[0] rows and r raw_data.shape[1] columns. Let’s inspect the first few entries across the initial six columns:

raw_data.iloc[:5, :6]
Response ID Q1_Student ID Q2_Age Q3_Gender Q4_Faculty Q5_Programme
0 R0001 TEST 99.0 Female NaN NaN
1 R0002 test NaN NaN NaN NaN
2 R0003 TEST2 30.0 Male NaN NaN
3 R0004 S0191 27.0 male Humanities Masters
4 R0005 S0015 30.0 Male Health Sciences Master's

Notice the irregularities typical of real-world research data: test records (TEST), unstandardized text variations (Male, male), and arbitrary missing value codes (99 for missing age). In ?@sec-ch03, we walk through systematic cleaning pipelines to sanitize this raw export.

If an Excel workbook contains multiple worksheets, you can target a specific sheet using the sheet_name argument (e.g., pd.read_excel("study.xlsx", sheet_name="Semester 2")).

2.6.3 SPSS data files

Quantitative social scientists frequently receive archives from colleagues formatted as SPSS files (.sav). The pyreadstat package enables Python to read .sav files while preserving variable metadata and response value labels:

import pyreadstat

spss_path = dt.example_file("wellbeing.sav")
spss_df, meta = pyreadstat.read_sav(spss_path)

spss_df.loc[:2, ["student_id", "gender", "faculty", "stress_1"]]
student_id gender faculty stress_1
0 S0001 1.0 2.0 3.0
1 S0002 1.0 1.0 4.0
2 S0003 2.0 1.0 4.0

SPSS files store underlying survey questions as variable labels. pyreadstat extracts these labels into the meta object:

print("Question text for stress_1:", meta.column_names_to_labels["stress_1"])
Question text for stress_1: I feel unable to control important things in my studies.

Alternatively, if you wish to import an SPSS file directly with numeric category codes decoded into human-readable text categories, Pandas provides pd.read_spss():

spss_labeled = pd.read_spss(spss_path)
spss_labeled["faculty"].value_counts()
faculty
Health Sciences     154
Education           148
Social Sciences     116
Humanities           95
Natural Sciences     87
Name: count, dtype: int64

This seamless interoperability ensures that years of meticulous labeling in SPSS are instantly accessible inside Python.

2.7 Automated codebook generation

In an empirical dataset, a cryptic column header like stress_1 or a numeric value like 4 is scientifically uninterpretable on its own. It acquires meaning only when accompanied by documentation specifying the original prompt (“I feel unable to control important things in my studies”), the response options (1 = Strongly Disagree to 5 = Strongly Agree), and missing data conventions. A codebook provides this reference, serving as the bridge between participant questionnaires and statistical analysis.

Because pyreadstat parses embedded SPSS metadata, we can programmatically assemble a comprehensive codebook DataFrame:

codebook = pd.DataFrame({
    "variable": meta.column_names,
    "label": meta.column_labels,
    "dtype": [str(spss_df[col].dtype) for col in meta.column_names]
})
codebook.head(10)
variable label dtype
0 student_id Student identifier str
1 supervisor_id Supervisor identifier str
2 age Age in years at baseline float64
3 gender Gender float64
4 faculty Faculty float64
5 programme Degree programme float64
6 study_mode Study mode float64
7 employment Paid work alongside study float64
8 has_children Has children float64
9 lives_away Moved away from family to study float64

For categorical items, the mapping between underlying integer codes and verbal labels is preserved in meta.variable_value_labels:

print("Value labels for faculty:", meta.variable_value_labels.get("faculty"))
Value labels for faculty: {1.0: 'Education', 2.0: 'Health Sciences', 3.0: 'Humanities', 4.0: 'Natural Sciences', 5.0: 'Social Sciences'}

Exporting this table into an appendix provides an automated, error-free documentation trail for your thesis committee and future secondary analysts.

2.8 Exporting data

Once raw data has been cleaned, validated, or transformed, the resulting DataFrame can be exported into standard file formats:

# Export to CSV (suppressing automatic integer index column)
students.to_csv("students_clean.csv", index=False)

# Export to Excel workbook
students.to_excel("students_clean.xlsx", index=False)

# Export to SPSS system file with pyreadstat
import pyreadstat
pyreadstat.write_sav(students, "students_clean.sav")

Setting index=False is essential when saving CSV and Excel files; otherwise, Pandas inserts an unlabelled leading column of row numbers into the file.

[!IMPORTANT] Preserve Raw Data Immutability: Never overwrite primary data files. Save processed datasets under descriptive filenames (e.g., students_clean.csv), ensuring that your Python script can recreate the clean data from raw inputs at any time.

2.9 Mapping SPSS and Excel workflows to Python

Moving from point-and-click tools to scripted programming involves mapping familiar conceptual operations to Python commands (Table 2.2).

Table 2.2: Conceptual mapping between point-and-click software and Python
Analytical Task SPSS / Excel Procedure Python / Pandas Idiom
Open data file File > Open > Data pd.read_csv(), pd.read_excel(), pd.read_spss()
Variable View (types & labels) Variable View tab df.info(), df.dtypes, meta.column_names_to_labels
Inspect data matrix Data View / Spreadsheet grid df.head(), VS Code Interactive Data Viewer
Select cases (filtering) Data > Select Cases df[condition] or df.loc[condition, :]
Compute variable Transform > Compute Variable df["new_col"] = df["a"] + df["b"]
Frequency distribution Analyze > Descriptive > Frequencies df["var"].value_counts()
Descriptive statistics Analyze > Descriptive > Descriptives df["var"].describe(), df["var"].mean()
Export data File > Save As df.to_csv(..., index=False), df.to_excel(...)

The paramount advantage of Python is transparency. In Excel or SPSS Data View, manual edits and filter clicks are transient and undocumented. In Python, the input dataset remains untouched while your script provides an indelible, auditable record of every transformation.

2.10 Reading error messages and tracebacks

Python tracebacks are diagnostic reports explaining precisely where and why execution halted. When an error occurs, scroll to the final line of the traceback to read the specific exception type and description. Four exceptions account for the vast majority of beginner stumbling blocks:

1. NameError: name 'slep' is not defined A variable name is misspelled, or the code defining it was not executed prior to this line:

print(slep)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[33], line 1
----> 1 print(slep)

NameError: name 'slep' is not defined

2. AttributeError or KeyError Referencing a non-existent column name inside brackets raises a KeyError (often caused by capitalisation mismatches):

students["Faculty"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/pandas/core/indexes/base.py:3816, in Index.get_loc(self, key)
   3815 try:
-> 3816     return self._engine.get_loc(casted_key)
   3817 except KeyError as err:

File pandas/_libs/index.pyx:172, in pandas._libs.index.IndexEngine.get_loc()
--> 172 'Could not get source, probably due dynamically evaluated source code.'

File pandas/_libs/index.pyx:201, in pandas._libs.index.IndexEngine.get_loc()
--> 201 'Could not get source, probably due dynamically evaluated source code.'

File pandas/_libs/hashtable_class_helper.pxi:7726, in pandas._libs.hashtable.PyObjectHashTable.get_item()
-> 7726 'Could not get source, probably due dynamically evaluated source code.'

File pandas/_libs/hashtable_class_helper.pxi:7734, in pandas._libs.hashtable.PyObjectHashTable.get_item()
-> 7734 'Could not get source, probably due dynamically evaluated source code.'

KeyError: 'Faculty'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[34], line 1
----> 1 students["Faculty"]

File /opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/pandas/core/frame.py:4330, in DataFrame.__getitem__(self, key)
   4326 
   4327         if is_single_key:
   4328             if self.columns.nlevels > 1:
   4329                 return self._getitem_multilevel(key)
-> 4330             indexer = self.columns.get_loc(key)
   4331             if is_integer(indexer):
   4332                 indexer = [indexer]  # type: ignore[assignment]
   4333         else:

File /opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/pandas/core/indexes/base.py:3823, in Index.get_loc(self, key)
   3818     if isinstance(casted_key, slice) or (
   3819         isinstance(casted_key, abc.Iterable)
   3820         and any(isinstance(x, slice) for x in casted_key)
   3821     ):
   3822         raise InvalidIndexError(key) from err
-> 3823     raise KeyError(key) from err
   3824 except TypeError:
   3825     # If we have a listlike key, _check_indexing_error will raise
   3826     #  InvalidIndexError. Otherwise we fall through and re-raise
   3827     #  the TypeError.
   3828     self._check_indexing_error(key)

KeyError: 'Faculty'

3. TypeError: unsupported operand type(s) Attempting mathematical operations on incompatible data types (for example, multiplying text strings):

"6.5" * "2"
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[35], line 1
----> 1 "6.5" * "2"

TypeError: can't multiply sequence by non-int of type 'str'

4. FileNotFoundError: No such file or directory Python searched the current working directory but could not locate the target filename. Verify your relative path against the folder opened in VS Code.

2.11 Descriptive summary of cohort measurements

With the data structures in place, Pandas provides .describe() for a rapid multi-variable statistical overview of central tendency, dispersion, and range:

first_semester[
    ["gpa", "sleep_hours", "study_hours", "wellbeing"]
].describe().round(2)
gpa sleep_hours study_hours wellbeing
count 600.00 594.00 592.00 600.00
mean 3.10 6.48 27.64 60.46
std 0.33 1.03 13.06 12.01
min 2.17 3.50 2.00 24.00
25% 2.89 5.80 18.00 52.00
50% 3.10 6.50 26.00 60.00
75% 3.34 7.20 37.00 69.00
max 4.00 10.00 67.00 93.00

This single command confirms that in semester 1, students slept an average of 6.48 hours (standard deviation = 0.94), well below recommended benchmarks. In ?@sec-ch06, we expand these basic summaries into publication-ready descriptive tables.

NoteIn your field: health research

The penguins dataset bundled with seaborn provides a classic biological sample across 344 observations and 7 variables:

import seaborn as sns

penguins = sns.load_dataset("penguins")
penguins.info()
penguins[["bill_length_mm", "bill_depth_mm", "body_mass_g"]].describe().round(1)
<class 'pandas.DataFrame'>
RangeIndex: 344 entries, 0 to 343
Data columns (total 7 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   species            344 non-null    str    
 1   island             344 non-null    str    
 2   bill_length_mm     342 non-null    float64
 3   bill_depth_mm      342 non-null    float64
 4   flipper_length_mm  342 non-null    float64
 5   body_mass_g        342 non-null    float64
 6   sex                333 non-null    str    
dtypes: float64(4), str(3)
memory usage: 18.9 KB
bill_length_mm bill_depth_mm body_mass_g
count 342.0 342.0 342.0
mean 43.9 17.2 4201.8
std 5.5 2.0 802.0
min 32.1 13.1 2700.0
25% 39.2 15.6 3550.0
50% 44.4 17.3 4050.0
75% 48.5 18.7 4750.0
max 59.6 21.5 6300.0

Notice the structural equivalence to human health studies: each row is a measured organism, categorical variables represent species and islands, and numeric metrics capture physiological dimensions.

2.12 Chapter review

2.12.1 Summary

  • Quantitative data is organized as a rectangular matrix where rows represent cases and columns represent variables. The unit of analysis defines what a single row represents (students vs repeated semesters).
  • A Pandas Series holds values of a single variable. Operations are vectorized, executing across all elements simultaneously without explicit loops.
  • Python uses 0-based indexing. Positional extraction relies on .iloc[], while conditional filtering relies on boolean masks series[condition].
  • Categories are managed via the Pandas category dtype, which supports explicit hierarchical ordering for ordinal scales.
  • A DataFrame is a two-dimensional tabular grid composed of individual Series. Subsetting rows and columns simultaneously is performed with .loc[rows, cols].
  • Data is imported using pd.read_csv(), pd.read_excel(), and pyreadstat.read_sav(). SPSS survey question labels and value encodings are preserved in metadata.
  • An automated codebook extracts column names, survey prompts, and data types to document study variables systematically.

2.12.2 Key terms

Data matrix, unit of analysis, Pandas Series, vectorization, boolean indexing, 0-based indexing, .iloc[], .loc[], categorical dtype, ordinal scale, DataFrame, correlation matrix, dictionary, CSV, SPSS metadata, codebook, traceback.

2.13 Exercises

Work through these exercises in your browser using the Chapter 2 Playground or download the starter project for VS Code.

  1. Create a Series with participant ages: 24, 31, 28, 45, 26. Use boolean indexing to filter ages over 30, and compute how many participants exceed 30.
  2. Construct an ordered Categorical Series from ["Agree", "Disagree", "Neutral", "Agree"] with categories ordered logically from "Disagree" to "Neutral" to "Agree". Verify the ordering using .value_counts(sort=False).
  3. In students, use boolean indexing with & to count how many participants study part-time (study_mode == "Part-time") AND have dependent children (children == "Yes").
  4. Add a boolean column over_30 to students that evaluates to True for students older than 30. Check how many missing values exist using .isna().sum().
  5. Import wellbeing.sav with pyreadstat and extract the full survey question text mapped to the variable support_3.
  6. Contrast the unit of analysis in students versus counselling_visits. Formulate one distinct research question appropriate for each dataset.

2.14 Further reading

  • Python for Data Analysis (3rd ed.) by Wes McKinney (McKinney 2022): the authoritative guide to Pandas indexing, Series arithmetic, and data wrangling by the creator of Pandas.
  • Official documentation for pandas.pydata.org and pyreadstat.

References

McKinney, Wes. 2022. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and Jupyter. 3rd ed. O’Reilly Media. https://wesmckinney.com/book/.