Data is the recorded outcome of scientific measurement. Each value captures one observation about one entity: a student’s age, the faculty they belong to, the hours they slept during a semester, or their response to a Likert questionnaire item. Before conducting any statistical analysis, a researcher must clarify what each value signifies, what level of measurement generated it, and how the data is organized in computer memory. These foundations determine which summary statistics, visualizations, and inferential models are scientifically valid.
Research data rarely arrives in a single pristine file. In Elaf’s wellbeing study, the online survey software generated an Excel export (wellbeing_raw.xlsx), the university registrar exported semester records as CSV files (semesters.csv), and a collaborator from a previous pilot study stored historical records in SPSS (wellbeing.sav). Before analyzing these heterogeneous sources, we need to understand how Python organizes data in memory. Python provides specialized structures tailored for data science: lists and NumPy arrays for basic sequences, Pandas Series for single measurement variables, Categorical Series for discrete groups, and DataFrames for rectangular tables of cases. This chapter introduces these structures, demonstrates how to import files from every common format, and shows how to generate an automated codebook linking every column to its underlying survey question.
TipBy the end of this chapter you will be able to
Conceptualize data as a rectangular matrix of cases and variables, and identify the unit of analysis.
Create Pandas Series, perform vectorized arithmetic, and filter observations with boolean indexing.
Manage categorical and ordinal measurements using Pandas Categorical types.
Construct, inspect, and slice two-dimensional DataFrames using .loc and .iloc.
Understand dictionaries and 2D matrices when encountered in statistical output.
Import real-world data from CSV, Excel (.xlsx), and SPSS (.sav) files, preserving survey question labels.
Build an automated codebook that documents variable definitions and measurement scales.
Map familiar SPSS and Excel procedures to Python idioms, and debug common Python error tracebacks.
Almost all empirical research data can be conceptualized as a rectangular grid: each row represents a case (or observation), and each column represents a variable. A case is the individual unit under observation—a student, a patient, an experimental plot, or an institution. A variable is a measured property recorded across all cases, such as chronological age, anxiety score, or study hours. Every cell in the grid contains a single value representing the measurement of that variable for that specific case. This structure, known as the data matrix, forms the backbone of statistical modeling.
What constitutes a case depends on the research design, defining the unit of analysis. Elaf’s wellbeing study involves multiple units of analysis across different tables. In the students table, each row represents an individual student (\(N = 600\)). In the semesters table, however, each row represents a single student in a specific academic semester, meaning each student appears up to four times:
semesters.head(5)
student_id
semester
gpa
sleep_hours
study_hours
exercise_days
caffeine_mg
supervisor_meetings
wellbeing
0
S0001
1
3.02
7.2
26.0
5.0
130.0
0
74
1
S0001
2
3.27
8.4
17.0
3.0
80.0
1
69
2
S0001
3
3.17
7.8
25.0
7.0
105.0
3
71
3
S0001
4
3.00
7.2
28.0
4.0
145.0
2
74
4
S0002
1
3.25
6.4
32.0
2.0
210.0
3
60
The first four rows all correspond to student S0001, documenting their trajectory across semesters 1 through 4. Confusing the unit of analysis is one of the most frequent errors in quantitative research. A demographic question concerning baseline enrollment must count each student exactly once. Conversely, an inquiry into GPA trajectories across time requires repeated-measures observations, coupled with statistical techniques that account for within-subject dependency (?@sec-ch10). In ?@sec-ch05, we explore how research hypotheses dictate the appropriate unit of analysis.
2.2 Variables and their values
In Python’s data science ecosystem, a single variable’s values are held in a Pandas Series (or a NumPy 1D array). A Series is an indexed, one-dimensional array capable of holding numbers, text, or boolean flags:
sleep = pd.Series([6.5, 7.0, 5.5, 8.0, 6.0])sleep
0 6.5
1 7.0
2 5.5
3 8.0
4 6.0
dtype: float64
A Series enforces a single homogeneous data type (dtype). If you inadvertently mix numbers with text, Pandas converts the entire Series to an object (or string) dtype to accommodate the text:
mixed = pd.Series([6.5, "seven", 5.5])mixed
0 6.5
1 seven
2 5.5
dtype: object
Notice that the numbers 6.5 and 5.5 have become text strings. They can no longer be averaged or plotted numerically. This behavior is vital to understand: if a single survey respondent types "seven" or "N/A" into a numeric field in a web form, the entire column imports into Python as text. We will encounter this exact issue when inspecting the raw Excel export later in this chapter.
2.2.1 Operations on a whole Series (Vectorization)
A defining strength of Pandas is vectorization: mathematical and logical operations apply across every value in the collection simultaneously, without needing explicit for loops. Converting sleep durations from hours to minutes requires a single multiplication:
Logical comparisons work identically, generating a boolean Series containing True or False for each entry. For example, identifying nights falling short of the clinical recommendation of 7 hours:
sleep <7
0 True
1 False
2 True
3 False
4 True
dtype: bool
Because Python evaluates True as 1 and False as 0 in numerical contexts, calling .sum() counts the qualifying nights, and .mean() calculates their proportion:
print("Count of short nights:", (sleep <7).sum())print("Proportion of short nights:", (sleep <7).mean())
Count of short nights: 3
Proportion of short nights: 0.6
Three of the five nights (60%) were short. This vectorized code operates at identical speed whether analyzing 5 observations or 500,000 observations.
2.2.2 Selecting values
Python uses 0-based indexing: the first element resides at position 0, the second at position 1, and so on. Think of an index as the offset or distance from the start of the array.
To select values by positional integer index in a Pandas Series, use .iloc[]:
print("First night:", sleep.iloc[0])print("First and third nights:\n", sleep.iloc[[0, 2]])
First night: 6.5
First and third nights:
0 6.5
2 5.5
dtype: float64
Values can also be extracted using a boolean condition inside square brackets. Pandas filters the Series, retaining only observations where the condition evaluates to True:
sleep[sleep <7]
0 6.5
2 5.5
4 6.0
dtype: float64
This syntax, termed boolean indexing (or logical filtering), reads naturally: “give me the elements of sleep where sleep is less than 7.” It is one of the most widely used idioms in quantitative Python.
2.2.3 Changing values
Placing an indexed selection on the left side of the assignment operator updates the underlying data. If a participant later clarifies that their second night was 7.5 hours rather than 7.0:
sleep.iloc[1] =7.5sleep
0 6.5
1 7.5
2 5.5
3 8.0
4 6.0
dtype: float64
2.3 Categories (Categorical Data)
Research datasets frequently contain qualitative categories: faculty division, experimental arm, gender identity, or Likert response levels. Pandas represents categorical variables using the specialized category dtype (similar to categorical variables in statistics or value labels in SPSS). A categorical Series displays text labels but stores them internally as compact integers mapped to a defined list of categories:
By default, categories are arranged in alphanumeric order. However, when categories possess an intrinsic hierarchy (such as employment intensity or Likert scales), you can define an ordered category using pd.Categorical:
Specifying explicit category orders ensures that subsequent summary tables and plots display categories logically from lowest to highest rather than alphabetically.
2.3.1 Levels of measurement and Python types
The distinction between continuous numbers and categories is not merely aesthetic; it reflects the underlying mathematical properties of the measurement. Categories without an order (such as faculty or academic discipline) are nominal. Categories with a strict ranking but unequal or unquantified intervals (such as Likert scales or employment tiers) are ordinal. Measurements with uniform, quantifiable units (such as sleep hours, age, or GPA) are interval or ratio (numeric). Table 2.1 outlines how each measurement level corresponds to a Python data type.
Table 2.1: Levels of measurement and their representation in Python
Measurement Level
Example Variables
Python Data Type
Nominal (unordered categories)
Faculty, Gender, Department
category (or object / string)
Ordinal (ordered categories)
Agreement scale, Education level
category (ordered=True)
Interval / Ratio (continuous quantities)
Sleep hours, Age, GPA, Scale sum
float64, int64
Binary / Dichotomous
Invited to workshop (Yes/No)
bool or two-level category
2.4 A table of cases: The DataFrame
Most empirical data is organized as a two-dimensional table where each row represents a case and each column represents a variable. In Python, this table is a Pandas DataFrame. Each column is a Series sharing a common index. While each column must have a uniform data type, different columns across the DataFrame can store different types.
You can construct a DataFrame manually from a Python dictionary:
The most comprehensive way to inspect a DataFrame’s architecture is the .info() method, which summarizes column names, non-null counts, and data types:
To slice rows and columns simultaneously, Pandas provides two primary indexers: 1. .iloc[row_pos, col_pos]: selects by integer position (0-based offset). 2. .loc[row_label, col_label]: selects by label name or boolean mask.
# Select the second row across all columnspilot.iloc[1, :]# Select all rows for two specific columnspilot.loc[:, ["student", "sleep"]]# Filter rows where sleep is under 7 hourspilot.loc[pilot["sleep"] <7, :]
student
faculty
sleep
invited
0
S1
Education
6.5
True
2
S3
Education
5.5
True
The final expression combines a DataFrame with boolean indexing: “retrieve all columns for rows where sleep is less than 7.” This is Python’s direct equivalent of SPSS’s Select Cases or Excel’s Filter.
2.4.2 Adding new columns
Assigning an expression to a new column name appends it to the DataFrame:
pilot["short_sleep"] = pilot["sleep"] <7pilot
student
faculty
sleep
invited
short_sleep
0
S1
Education
6.5
True
True
1
S2
Humanities
7.5
False
False
2
S3
Education
5.5
True
True
3
S4
Health Sciences
8.0
False
False
2.4.3 The student wellbeing cohort
The students dataset bundled in data2thesis follows this exact DataFrame architecture across 600 participants:
In this output, object indicates text strings, int64 denotes integers, and float64 denotes decimal numbers. Columns representing categories, such as faculty, can be converted to categorical types whenever discrete factor levels are needed:
Subsetting with multiple criteria uses Python’s bitwise logical operators: & (AND) and | (OR). In Pandas, each individual condition must be enclosed in parentheses () because bitwise operators take precedence over comparison operators (==, >, <):
In VS Code, opening any DataFrame in the Interactive Window or Data Viewer provides an interactive spreadsheet grid where you can sort, filter, and inspect columns visually without altering the underlying code.
2.5 Matrices and Dictionaries
Beyond Series and DataFrames, two additional data structures frequently appear in quantitative workflows: 2D numerical matrices and dictionaries.
2.5.1 Matrices and correlation grids
A numerical matrix is a two-dimensional grid of homogeneous numbers. Matrices arise naturally as the mathematical output of statistical procedures, such as calculating bivariate correlation coefficients across multiple metrics:
Each variable correlates perfectly with itself (\(r = 1.00\)) along the main diagonal. Specific pairwise correlations can be extracted using .loc[row, col]:
correlations.loc["sleep_hours", "wellbeing"]
np.float64(0.45630448626446257)
2.5.2 Dictionaries and statistical results
A dictionary (dict) is Python’s fundamental key-value container. It stores named attributes enclosed in curly braces {}:
While researchers rarely build complex dictionaries by hand, many Python scientific packages return statistical test results as structured objects or dictionary-like containers. For instance, running a one-sample \(t\)-test with scipy.stats to test whether first-semester students sleep 7 hours on average:
The resulting object encapsulates the computed test statistic, degrees of freedom, and \(p\)-value as named attributes ready for automated reporting in your thesis.
2.6 Importing research data
Different software packages store tabular data in distinct binary or text formats. Python provides dedicated reader functions for every standard format, each parsing the input directly into a clean Pandas DataFrame. The sample files below are installed with data2thesis and located via dt.example_file():
2.6.1 CSV files
A CSV file (comma-separated values) is a plain-text table where commas delimit adjacent columns. Because CSV files are universally supported and human-readable, they represent the gold standard for long-term open science archiving. In Python, CSV files are imported using pd.read_csv():
In your own thesis project, you will typically point directly to your project’s data directory: pd.read_csv("data/semesters.csv").
2.6.2 Excel spreadsheets
Real-world survey platforms frequently export multi-column workbooks (.xlsx). Pandas reads modern Excel files via pd.read_excel() (powered by the openpyxl engine):
This file represents an unfiltered export from the wellbeing survey. The .shape property indicates r raw_data.shape[0] rows and r raw_data.shape[1] columns. Let’s inspect the first few entries across the initial six columns:
raw_data.iloc[:5, :6]
Response ID
Q1_Student ID
Q2_Age
Q3_Gender
Q4_Faculty
Q5_Programme
0
R0001
TEST
99.0
Female
NaN
NaN
1
R0002
test
NaN
NaN
NaN
NaN
2
R0003
TEST2
30.0
Male
NaN
NaN
3
R0004
S0191
27.0
male
Humanities
Masters
4
R0005
S0015
30.0
Male
Health Sciences
Master's
Notice the irregularities typical of real-world research data: test records (TEST), unstandardized text variations (Male, male), and arbitrary missing value codes (99 for missing age). In ?@sec-ch03, we walk through systematic cleaning pipelines to sanitize this raw export.
If an Excel workbook contains multiple worksheets, you can target a specific sheet using the sheet_name argument (e.g., pd.read_excel("study.xlsx", sheet_name="Semester 2")).
2.6.3 SPSS data files
Quantitative social scientists frequently receive archives from colleagues formatted as SPSS files (.sav). The pyreadstat package enables Python to read .sav files while preserving variable metadata and response value labels:
SPSS files store underlying survey questions as variable labels. pyreadstat extracts these labels into the meta object:
print("Question text for stress_1:", meta.column_names_to_labels["stress_1"])
Question text for stress_1: I feel unable to control important things in my studies.
Alternatively, if you wish to import an SPSS file directly with numeric category codes decoded into human-readable text categories, Pandas provides pd.read_spss():
faculty
Health Sciences 154
Education 148
Social Sciences 116
Humanities 95
Natural Sciences 87
Name: count, dtype: int64
This seamless interoperability ensures that years of meticulous labeling in SPSS are instantly accessible inside Python.
2.7 Automated codebook generation
In an empirical dataset, a cryptic column header like stress_1 or a numeric value like 4 is scientifically uninterpretable on its own. It acquires meaning only when accompanied by documentation specifying the original prompt (“I feel unable to control important things in my studies”), the response options (1 = Strongly Disagree to 5 = Strongly Agree), and missing data conventions. A codebook provides this reference, serving as the bridge between participant questionnaires and statistical analysis.
Because pyreadstat parses embedded SPSS metadata, we can programmatically assemble a comprehensive codebook DataFrame:
codebook = pd.DataFrame({"variable": meta.column_names,"label": meta.column_labels,"dtype": [str(spss_df[col].dtype) for col in meta.column_names]})codebook.head(10)
variable
label
dtype
0
student_id
Student identifier
str
1
supervisor_id
Supervisor identifier
str
2
age
Age in years at baseline
float64
3
gender
Gender
float64
4
faculty
Faculty
float64
5
programme
Degree programme
float64
6
study_mode
Study mode
float64
7
employment
Paid work alongside study
float64
8
has_children
Has children
float64
9
lives_away
Moved away from family to study
float64
For categorical items, the mapping between underlying integer codes and verbal labels is preserved in meta.variable_value_labels:
print("Value labels for faculty:", meta.variable_value_labels.get("faculty"))
Value labels for faculty: {1.0: 'Education', 2.0: 'Health Sciences', 3.0: 'Humanities', 4.0: 'Natural Sciences', 5.0: 'Social Sciences'}
Exporting this table into an appendix provides an automated, error-free documentation trail for your thesis committee and future secondary analysts.
2.8 Exporting data
Once raw data has been cleaned, validated, or transformed, the resulting DataFrame can be exported into standard file formats:
# Export to CSV (suppressing automatic integer index column)students.to_csv("students_clean.csv", index=False)# Export to Excel workbookstudents.to_excel("students_clean.xlsx", index=False)# Export to SPSS system file with pyreadstatimport pyreadstatpyreadstat.write_sav(students, "students_clean.sav")
Setting index=False is essential when saving CSV and Excel files; otherwise, Pandas inserts an unlabelled leading column of row numbers into the file.
[!IMPORTANT] Preserve Raw Data Immutability: Never overwrite primary data files. Save processed datasets under descriptive filenames (e.g., students_clean.csv), ensuring that your Python script can recreate the clean data from raw inputs at any time.
2.9 Mapping SPSS and Excel workflows to Python
Moving from point-and-click tools to scripted programming involves mapping familiar conceptual operations to Python commands (Table 2.2).
Table 2.2: Conceptual mapping between point-and-click software and Python
Analytical Task
SPSS / Excel Procedure
Python / Pandas Idiom
Open data file
File > Open > Data
pd.read_csv(), pd.read_excel(), pd.read_spss()
Variable View (types & labels)
Variable View tab
df.info(), df.dtypes, meta.column_names_to_labels
Inspect data matrix
Data View / Spreadsheet grid
df.head(), VS Code Interactive Data Viewer
Select cases (filtering)
Data > Select Cases
df[condition] or df.loc[condition, :]
Compute variable
Transform > Compute Variable
df["new_col"] = df["a"] + df["b"]
Frequency distribution
Analyze > Descriptive > Frequencies
df["var"].value_counts()
Descriptive statistics
Analyze > Descriptive > Descriptives
df["var"].describe(), df["var"].mean()
Export data
File > Save As
df.to_csv(..., index=False), df.to_excel(...)
The paramount advantage of Python is transparency. In Excel or SPSS Data View, manual edits and filter clicks are transient and undocumented. In Python, the input dataset remains untouched while your script provides an indelible, auditable record of every transformation.
2.10 Reading error messages and tracebacks
Python tracebacks are diagnostic reports explaining precisely where and why execution halted. When an error occurs, scroll to the final line of the traceback to read the specific exception type and description. Four exceptions account for the vast majority of beginner stumbling blocks:
1. NameError: name 'slep' is not defined A variable name is misspelled, or the code defining it was not executed prior to this line:
print(slep)
---------------------------------------------------------------------------NameError Traceback (most recent call last)
CellIn[33], line 1----> 1 print(slep)
NameError: name 'slep' is not defined
2. AttributeError or KeyError Referencing a non-existent column name inside brackets raises a KeyError (often caused by capitalisation mismatches):
students["Faculty"]
---------------------------------------------------------------------------KeyError Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/pandas/core/indexes/base.py:3816, in Index.get_loc(self, key) 3815try:
-> 3816returnself._engine.get_loc(casted_key) 3817exceptKeyErroras err:
File pandas/_libs/index.pyx:172, in pandas._libs.index.IndexEngine.get_loc()--> 172'Could not get source, probably due dynamically evaluated source code.'File pandas/_libs/index.pyx:201, in pandas._libs.index.IndexEngine.get_loc()--> 201'Could not get source, probably due dynamically evaluated source code.'File pandas/_libs/hashtable_class_helper.pxi:7726, in pandas._libs.hashtable.PyObjectHashTable.get_item()-> 7726'Could not get source, probably due dynamically evaluated source code.'File pandas/_libs/hashtable_class_helper.pxi:7734, in pandas._libs.hashtable.PyObjectHashTable.get_item()-> 7734'Could not get source, probably due dynamically evaluated source code.'KeyError: 'Faculty'
The above exception was the direct cause of the following exception:
KeyError Traceback (most recent call last)
CellIn[34], line 1----> 1 students["Faculty"]
File /opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/pandas/core/frame.py:4330, in DataFrame.__getitem__(self, key) 4326 4327if is_single_key:
4328if self.columns.nlevels > 1:
4329return self._getitem_multilevel(key)
-> 4330 indexer = self.columns.get_loc(key)
4331if is_integer(indexer):
4332 indexer = [indexer] # type: ignore[assignment] 4333else:
File /opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/pandas/core/indexes/base.py:3823, in Index.get_loc(self, key) 3818ifisinstance(casted_key, slice) or (
3819isinstance(casted_key, abc.Iterable)
3820andany(isinstance(x, slice) for x in casted_key)
3821 ):
3822raise InvalidIndexError(key) fromerr-> 3823raiseKeyError(key) fromerr 3824exceptTypeError:
3825# If we have a listlike key, _check_indexing_error will raise 3826# InvalidIndexError. Otherwise we fall through and re-raise 3827# the TypeError. 3828self._check_indexing_error(key)
KeyError: 'Faculty'
3. TypeError: unsupported operand type(s) Attempting mathematical operations on incompatible data types (for example, multiplying text strings):
"6.5"*"2"
---------------------------------------------------------------------------TypeError Traceback (most recent call last)
CellIn[35], line 1----> 1"6.5" * "2"TypeError: can't multiply sequence by non-int of type 'str'
4. FileNotFoundError: No such file or directory Python searched the current working directory but could not locate the target filename. Verify your relative path against the folder opened in VS Code.
2.11 Descriptive summary of cohort measurements
With the data structures in place, Pandas provides .describe() for a rapid multi-variable statistical overview of central tendency, dispersion, and range:
This single command confirms that in semester 1, students slept an average of 6.48 hours (standard deviation = 0.94), well below recommended benchmarks. In ?@sec-ch06, we expand these basic summaries into publication-ready descriptive tables.
NoteIn your field: health research
The penguins dataset bundled with seaborn provides a classic biological sample across 344 observations and 7 variables:
import seaborn as snspenguins = sns.load_dataset("penguins")penguins.info()penguins[["bill_length_mm", "bill_depth_mm", "body_mass_g"]].describe().round(1)
Notice the structural equivalence to human health studies: each row is a measured organism, categorical variables represent species and islands, and numeric metrics capture physiological dimensions.
2.12 Chapter review
2.12.1 Summary
Quantitative data is organized as a rectangular matrix where rows represent cases and columns represent variables. The unit of analysis defines what a single row represents (students vs repeated semesters).
A Pandas Series holds values of a single variable. Operations are vectorized, executing across all elements simultaneously without explicit loops.
Python uses 0-based indexing. Positional extraction relies on .iloc[], while conditional filtering relies on boolean masks series[condition].
Categories are managed via the Pandas category dtype, which supports explicit hierarchical ordering for ordinal scales.
A DataFrame is a two-dimensional tabular grid composed of individual Series. Subsetting rows and columns simultaneously is performed with .loc[rows, cols].
Data is imported using pd.read_csv(), pd.read_excel(), and pyreadstat.read_sav(). SPSS survey question labels and value encodings are preserved in metadata.
An automated codebook extracts column names, survey prompts, and data types to document study variables systematically.
2.12.2 Key terms
Data matrix, unit of analysis, Pandas Series, vectorization, boolean indexing, 0-based indexing, .iloc[], .loc[], categorical dtype, ordinal scale, DataFrame, correlation matrix, dictionary, CSV, SPSS metadata, codebook, traceback.
2.13 Exercises
Work through these exercises in your browser using the Chapter 2 Playground or download the starter project for VS Code.
Create a Series with participant ages: 24, 31, 28, 45, 26. Use boolean indexing to filter ages over 30, and compute how many participants exceed 30.
Construct an ordered Categorical Series from ["Agree", "Disagree", "Neutral", "Agree"] with categories ordered logically from "Disagree" to "Neutral" to "Agree". Verify the ordering using .value_counts(sort=False).
In students, use boolean indexing with & to count how many participants study part-time (study_mode == "Part-time") AND have dependent children (children == "Yes").
Add a boolean column over_30 to students that evaluates to True for students older than 30. Check how many missing values exist using .isna().sum().
Import wellbeing.sav with pyreadstat and extract the full survey question text mapped to the variable support_3.
Contrast the unit of analysis in students versus counselling_visits. Formulate one distinct research question appropriate for each dataset.
2.14 Further reading
Python for Data Analysis (3rd ed.) by Wes McKinney (McKinney 2022): the authoritative guide to Pandas indexing, Series arithmetic, and data wrangling by the creator of Pandas.