By the end of today you can

• explain how the reliability of coding is measured, and why chance matters;
• explain how large language models work, and why they make confident mistakes;
• use an AI assistant to write, explain, and debug Python code, and check its answers;
• call a language model from Python (via Ollama or APIs) with structured schemas;
• validate AI coding against human coding with accuracy and Cohen’s kappa;
• use AI responsibly: protect participant data, keep work reproducible, and disclose tools.

Text as research data

“What has been your biggest challenge during your studies?”

536 students answered in their own words (RQ11).

200 answers were coded by hand into themes, which took most of two weeks. Could an AI tool code the rest reliably?

Qualitative coding

Reading each text and assigning it to a theme defined in a codebook.

Coding is judgement, and judgements differ.

Whether the coder is a person, a keyword regex dictionary, or a machine learning model, the question is the same: does it code reliably?

Raw agreement can mislead

coder_a = ["Workload"] * 16 + ["Family"] * 4
coder_b = ["Workload"] * 14 + ["Family"] * 2 + ["Workload"] * 2 + ["Family"] * 2

Each coder puts 80% under Workload, so random coders matching their class distributions would agree 68% of the time purely by chance!

Cohen’s kappa

\[ \kappa = \frac{\text{observed} - \text{chance}}{1 - \text{chance}} \]

cohen_kappa_score() from scikit-learn. The standard measure of inter-rater reliability.

How large language models work

• a neural network (Chapter 15) with billions of parameters/weights;
• trained on vast corpora of text: books, research papers, websites, code;
• learns to predict the next token (a word or sub-word unit);
• answers by generating one likely token at a time;
• instruction-tuning (RLHF) shapes it into an assistant that follows prompts.

Why they make confident mistakes

Use AI to draft, and check everything it produces.

Asking good questions

A plausible answer that is wrong

students.groupby("faculty")["considering_dropout"].apply(
    lambda x: 100 * (x == "Yes").sum() / len(students)
)

It runs without errors, and the numbers look plausible.

But they add up to 21% across all faculties: len(students) divides by all 600 students, rather than the denominator of each faculty!

The corrected code

students.groupby("faculty")["considering_dropout"].apply(
    lambda x: 100 * (x == "Yes").mean()
)

The mean of a boolean Series (x == 'Yes') gives the proportion True within each faculty (Chapter 6).

Not a hallucination, but a plausible misunderstanding: the most dangerous kind of bug, because nothing crashes.

Four checks

• run the code on a tiny synthetic example with a known ground truth;
• check a number by another route: pd.crosstab(students['faculty'], students['considering_dropout'], normalize='index');
• inspect the documentation: does the argument or function actually do what you think?
• ask the assistant to explain each step line by line.

Assistants are superb tutors: “explain this line by line”, “why did this produce NaNs?”.

AI inside the editor

• GitHub Copilot in VS Code suggests code completions inline as you type;
• Claude / Cursor / Gemini can inspect workspace context and provide contextual edits.

Suggestions arrive instantly and look authoritative. Always read and verify each suggestion before accepting.

The answers to be coded

open_responses.iloc[[4, 8, 14, 54]]["biggest_challenge"].tolist()

They vary in length and style, often mention multiple challenges simultaneously, and contain typos.

Seven themes. The 200 hand-coded responses serve as the benchmark test set (Chapter 11).

The hand-coded themes

| Theme       |   Count |
|:------------|--------:|
| Supervision |      68 |
| Workload    |      45 |
| Isolation   |      31 |
| Health      |      20 |
| Finances    |      14 |
| Family      |      14 |
| Other       |       8 |

Coding by keywords

keywords = {
    "Supervision": r"\b(supervis|feedback|guidance|meeting)",
    "Finances":    r"\b(money|fee|scholarship|rent|afford|salary|income|funding)",
    "Family":      r"\b(famil|child|kid|son\b|daughter|baby|parent|husband|wife)"
}  # plus Workload, Health, Isolation, and Other

A regular expression: | means “or”, \b asserts a word boundary.

Words are not meaning

code_by_keywords("I miss my family and I cannot afford the fees.")
# Output: 'Finances'

One Family word, one Isolation phrase, two Finances words: Finances wins by count, even though a human reader might judge family or isolation paramount.

“My supervisor” mentioned in an answer about letting down one’s family is really about family.

Calling a language model from Python

Store API keys in environment variables or .env files, never hardcoded in scripts or pushed to Git.

A local model via Ollama

import ollama

client = ollama.Client()
response = client.chat(
    model="gemma4:e4b",
    messages=[{"role": "user", "content": "In one sentence, what does Cohen's kappa measure?"}],
    options={"temperature": 0.0}
)
print(response["message"]["content"])

Always specify the exact model identifier and set temperature: 0.0 for deterministic reproducibility.

The codebook as a system prompt

codebook = """
You are helping a researcher code open-ended survey answers ...
Assign each answer to exactly ONE theme: the main challenge.
Themes:
- Supervision: the supervisor; feedback, guidance, meetings.
- Workload: too much work or too little time; deadlines, reading.
- Finances: money, fees, scholarships, living costs.
- Family: children, partners, parents, caring duties.
- Health: physical or mental health, sleep, stress, burnout.
- Isolation: loneliness, far from home, no colleagues.
- Other: anything else, such as ethics approval or statistics.
"""

Consistent guidelines supplied to every evaluation call.

Structured outputs with Pydantic

from pydantic import BaseModel
from typing import Literal

ThemeLiteral = Literal["Supervision", "Workload", "Finances", "Family", "Health", "Isolation", "Other"]

class CodedAnswer(BaseModel):
    theme: ThemeLiteral
    confidence: float

Structured schemas force the model to output valid, strongly typed JSON fields rather than conversational chit-chat.

Checking replies in code

# Fallback parser for plain-text replies from compact local models:
def parse_theme(reply_text, valid_themes=themes):
    cleaned = re.sub(r"[^A-Za-z]", "", reply_text).lower()
    for t in valid_themes:
        if cleaned == t.lower():
            return t
    return np.nan

Any reply failing schema compliance produces NaN—which is audited and counted, rather than silently ignored.

Run once, save, analyse the file

The script records model checkpoint, execution timestamp, library versions, and elapsed runtime.

The model against the hand coding

The model achieves almost perfect agreement (\(\kappa = 0.874\)) with human ground truth, outperforming regex dictionaries by 29 percentage points.

Read the disagreements

| Answer                                                               | Hand Coded   | Model     |
|:---------------------------------------------------------------------|:-------------|:----------|
| Getting feedback on my draft took over a month, and by then I had... | Supervision  | Isolation |
| Combining coursework, my job at a school and my thesis leaves me ... | Workload     | Health    |
| I stopped exercising and I can feel the difference. The doctor to... | Health       | Other     |
| Writing my literature review took far longer than I planned. And ... | Workload     | Isolation |

Is the model wrong, the human coder inconsistent, or the text genuinely polysemous? Auditing disagreements clarifies codebook definitions.

Is the model repeatable?

Tested twice at temperature = 0.0, the model yielded 100% identical thematic classifications across all 200 responses.

Repeatability is essential for reproducible science: outputs are persisted to CSV and archived.

What the students said

Distribution of all 536 student responses, automated after kappa validation.

Text and numbers tell the same story

| Theme       |   Students |   Mean Supervisor Support |
|:------------|-----------:|--------------------------:|
| Supervision |        177 |                      2.82 |
| Isolation   |         61 |                      3.3  |
| Other       |         23 |                      3.31 |
| Health      |         56 |                      3.33 |
| Family      |         47 |                      3.43 |
| Finances    |         49 |                      3.45 |
| Workload    |        122 |                      3.46 |

Students classified under “Supervision” reported the lowest supervisor support scores on quantitative Likert items (RQ11 criterion validity).

Check, and stay responsible

• run generated code on synthetic validation cases;
• verify external citations: language models frequently fabricate non-existent DOI references;
• validate algorithmic coding against hand-coded benchmarks (\(\kappa \ge 0.80\)).

You remain personally and academically responsible for every claim, table, and analysis in your thesis.

Protect your data

• text transmitted to third-party APIs leaves your machine and may be logged or used for training;
• verify ethics committee approval and participant consent forms before any external processing;
• scrub names, emails, and identifying organizational details;
• for sensitive, confidential, or clinical data, deploy an offline local model (e.g. Ollama).

Keep it reproducible

Disclose it

AI tools cannot be listed as co-authors because they cannot assume legal or ethical responsibility (COPE / ICMJE guidelines).

“Open-ended survey responses were coded into seven themes using a local large language model (Gemma 4 e4b via Ollama, temperature 0.0) following the codebook in Appendix X. Inter-coder reliability was validated against a random hand-coded sample of 200 responses using Cohen’s kappa (\(\kappa = 0.874\)). Prompts, schemas, and raw outputs are archived in the supplementary materials.”

Be aware of bias

Models inherit systematic biases from training data:

• skewed framing across demographic subgroups;
• disparate performance on non-standard dialects or second-language English phrasing;
• cultural assumptions embedded in pre-training corpora.

Evaluate validation accuracy across demographic subsets to detect performance disparities.

In your field: medicine and health data

from pydantic import BaseModel

class MedicationExtraction(BaseModel):
    drug: str
    dose_mg: float
    frequency_per_day: int

Convert unstructured clinical notes into validated, structured records. Always validate against clinician-reviewed gold standards, and never upload patient data to cloud services.

Practical lab: Chapter 18 playground

Complete the interactive exercises in your browser:

• checking assistant scripts for silent logical errors;
• dictionary regex matching vs. machine learning;
• computing observed agreement, chance agreement, and Cohen’s kappa;
• auditing confusion matrices and disagreement patterns.

Completion checklist

• AI-generated code tested against boundary cases;
• Automated qualitative coding validated against human benchmarks (\(\kappa \ge 0.80\));
• Disagreements manually inspected and codebook refined;
• Data privacy respected (anonymization or local inference);
• Execution metadata (model, version, prompt, temperature) recorded;
• AI usage transparently disclosed in methodology section.

Misconceptions to leave behind

The chapter in one sentence

Treat AI output as a draft: validate it against human judgement, protect participants, record what you did, and say so.

Next: Chapter 19

The capstone chapter brings every component together into a completed thesis:

• end-to-end data pipelines from raw exports to tidy data;
• answering each research question systematically;
• building polished tables and figures;
• connecting findings into a coherent academic discussion.

Questions

Which part of your own analysis would you trust an AI to draft?

How would you check it?