Much research data is text: answers to open questions, interview transcripts, clinical notes, and policy documents. To analyse qualitative text quantitatively, researchers use qualitative coding: reading each text passage and assigning it to one of a set of themes defined in a systematic codebook. Coding is an act of judgement, and judgements differ. A central methodological question is therefore how to determine whether a coder—whether a human researcher, a keyword dictionary, or a computer model—codes reliably. The standard answer is to compare the coder with an independent benchmark, and to measure how far they agree beyond what chance alone would produce.
In the wellbeing study, the final survey included one open question: “What has been your biggest challenge during your studies?” A total of 536 students answered in their own words, and the eleventh research question (RQ11) investigates what those challenges are. Elaf coded 200 answers by hand, which required two full weeks of intense concentration. The methodological challenge is whether an AI tool could code the remaining answers reliably.
Artificial intelligence (AI) tools now play a major role across research: coding assistants that write and explain Python code, and large language models (LLMs) that read, summarize, and classify text. This chapter explores both applications. It explains how these models work and why they make confident errors (hallucinations), shows how to use an AI assistant to write robust Python code and how to verify its outputs, and then demonstrates how to call a language model from Python to code students’ responses, validating the results against human coding with accuracy and Cohen’s kappa. It concludes with essential ethical guidelines for using AI responsibly in scientific research.
TipBy the end of this chapter you will be able to
Explain how the reliability of coding is measured, and why raw agreement must be corrected for chance.
Explain in plain terms how large language models work, and why they make confident mistakes.
Use an AI assistant to write, explain, and debug Python code, and verify its suggestions.
Call language models from Python (online or local via Ollama) and obtain structured classifications for text datasets.
Validate AI qualitative coding against human benchmark coding using accuracy and Cohen’s kappa.
Use AI responsibly: protect participant data, preserve reproducibility, and disclose AI assistance.
Two coders who read the same answers will not always choose the same theme. Their raw agreement—the proportion of answers on which they assign identical themes—seems like the natural metric, but it can be highly deceptive. When one theme is very frequent, two coders will often agree simply because both choose it most of the time by chance.
A toy example illustrates why this matters. Suppose two coders categorize 20 student answers, and one coder places 16 of them under Workload:
The coders agree on 80% of the answers, which initially sounds strong. But because each coder classifies 80% of the responses under Workload, two coders guessing at random in those proportions would agree on 68% of the answers by chance alone!
Cohen’s kappa measures agreement beyond chance: the observed agreement minus chance agreement, divided by the maximum possible agreement above chance (Cohen 1960):
\[
\kappa = \frac{P_o - P_e}{1 - P_e}
\]
Here, Cohen’s kappa is only 0.38. A kappa of 0 represents chance-level agreement, and 1 indicates perfect agreement. Standard benchmarks (Landis and Koch 1977) classify \(\kappa\) as:
0.00 – 0.20: Slight agreement
0.21 – 0.40: Fair agreement
0.41 – 0.60: Moderate agreement
0.61 – 0.80: Substantial agreement
0.81 – 1.00: Almost perfect agreement
In Python, the cohen_kappa_score function from scikit-learn computes this directly:
from sklearn.metrics import cohen_kappa_scoreprint(f"scikit-learn kappa: {cohen_kappa_score(coder_a, coder_b):.2f}")
scikit-learn kappa: 0.38
Kappa is the standard measure of inter-rater reliability (Chapter 5), and it applies to any pair of coders—whether two human raters, a keyword dictionary, or a language model.
18.2 How AI assistants work
Modern AI coding assistants (ChatGPT, Claude, Gemini, GitHub Copilot) are powered by large language models (LLMs). An LLM is a deep neural network (Chapter 15) containing billions of parameters, trained on massive corpora of text and source code.
During training, the model learns to perform one core task: predict the next token (a word or sub-word piece) given the preceding context. Generating a response proceeds token by token, continually selecting statistically probable continuations. Instruction tuning and reinforcement learning from human feedback (RLHF) then shape the raw model into an assistant that follows conversational instructions.
This fundamental architecture explains both their strengths and their hazards:
Strengths: LLMs are remarkably fluent in Python, NumPy, pandas, and scikit-learn because extensive documentation, tutorials, and open-source code were present in their training data. They excel at writing initial boilerplate, translating concepts between languages, explaining obscure error traces, and suggesting alternative library functions.
Hazards: LLMs generate text that is statistically plausible, not factually verified. A non-existent method name, an obsolete argument, or a subtly incorrect statistical denominator can be generated with total stylistic confidence. Such fabrications are known as hallucinations. Furthermore, a model’s knowledge ceases at its training cut-off date, and it possesses no intrinsic awareness of your local data structures unless you provide them.
The practical rule is absolute: treat AI output as a draft, and verify every line yourself. You remain solely responsible for the integrity of your thesis.
18.3 AI as a coding assistant
18.3.1 Asking effective questions
An AI assistant’s response is only as precise as the prompt it receives. To obtain reliable Python code:
State the goal clearly: e.g., “For each faculty, compute the percentage of students who considered dropping out.”
Describe the data structure: provide column names and types (e.g., using students.dtypes or students.head(2)), ensuring no sensitive or identifiable participant data is ever pasted into online prompts.
Specify the preferred libraries: e.g., “Use pandas groupby and vectorised methods, avoiding deprecated syntax.”
Provide exact error messages: when debugging, paste the full traceback along with the offending code snippet.
18.3.2 Verifying the suggestion
Suppose an assistant is asked for the percentage of students in each faculty who considered dropping out, and suggests the following pandas code:
# Plausible but incorrect suggestion:flawed = ( students.groupby('faculty')['considering_dropout'] .apply(lambda s: 100* (s =='Yes').sum() /len(students)) .reset_index(name='percent'))flawed
faculty
percent
0
Education
2.666667
1
Health Sciences
4.833333
2
Humanities
2.166667
3
Natural Sciences
1.833333
4
Social Sciences
3.500000
The code runs without error and the numbers look believable. However, the percentages sum to only 15.7%—the percentage of all students who considered dropping out! Dividing by len(students) divides by the global sample size rather than the faculty subgroup size.
The correct formulation computes the proportion within each faculty:
Because (s == 'Yes') is a boolean series, its mean is the exact within-faculty proportion (Chapter 6). The assistant’s error was not an obvious crash; it was a subtle conceptual mismatch.
Catching such errors requires four standard habits:
Test on toy data: run code on a tiny 3-row DataFrame where you know the manual calculation.
Cross-check by alternative routes: cross-tabulate counts with pd.crosstab(students['faculty'], students['considering_dropout'], normalize='index').
Inspect library documentation: verify parameters and return signatures in official docstrings.
Ask the assistant to explain: requesting “explain how the denominator is computed” often exposes faulty logic immediately.
18.4 Coding open-ended survey answers
A selection of raw survey responses illustrates the challenge of open text:
for idx in [4, 8, 14, 54]:print(f"- \"{open_responses.loc[idx, 'biggest_challenge']}\"")
- "most of my income goes on tuition fees. Besides that, I work alone all the time. Things are slowly improving."
- "I moved to a city where I know no one for my studies and I feel lonely. And I never have enough time."
- "Supervisor."
- "The direction of my thesis changed twice after comments from my supervsor. Sometimes I rewrite the same section again and again without knowing if it is right."
Responses range from brief phrases to elaborate paragraphs, frequently mention multiple competing pressures, and feature informal grammar and typos. The qualitative codebook defines seven mutually exclusive themes: Supervision, Workload, Finances, Family, Health, Isolation, and Other.
Elaf hand-coded a random validation sample of 200 answers into the single primary challenge:
theme
Supervision 68
Workload 45
Isolation 31
Health 20
Finances 14
Family 14
Other 8
Name: count, dtype: int64
This 200-student dataset serves as the gold-standard test set against which automatic methods are benchmarked.
18.4.1 Coding with a keyword dictionary
The simplest automated baseline is a rule-based keyword dictionary. Each answer is assigned to the theme with the highest number of keyword matches, defaulting to “Other” if no keywords match:
keywords = {"Supervision": r"\b(supervis|feedback|guidance|meeting)","Workload": r"\b(time|deadline|workload|too much|assignment|reading|busy)","Finances": r"\b(money|fee|scholarship|rent|afford|salary|income|funding|stipend|loan)","Family": r"\b(famil|child|kid|son\b|daughter|baby|parent|mother|father|husband|wife)","Health": r"\b(sleep|tired|stress|anxi|health|ill\b|burnout|headache|coffee|exhaust)","Isolation": r"\b(lonel|alone|friends|outsider|miss (home|my)|far from|know no one)","Other": r"\b(ethic|participant|statistic|library|procedure|english|power cut|laptop|internet|software)"}themes =list(keywords.keys())def code_by_keywords(text): text_lower =str(text).lower() hits = {theme: len(re.findall(pat, text_lower)) for theme, pat in keywords.items()} max_hits =max(hits.values())if max_hits ==0:return"Other"for theme, count in hits.items():if count == max_hits:return themesample_text ="I miss my family and I cannot afford the fees."print(f"Sample coding: '{sample_text}' -> {code_by_keywords(sample_text)}")
Sample coding: 'I miss my family and I cannot afford the fees.' -> Finances
Evaluating the dictionary against Elaf’s 200 hand-coded responses:
The keyword dictionary achieves 61.0% accuracy and a Cohen’s kappa of 0.51 (moderate agreement). Keyword matching struggles because semantic meaning depends on syntactic context: the phrase “I feel I am letting down both my family and my supervisor” matches keywords in both Family and Supervision, while “I never have enough time” could reflect course workload or family caring duties. Discerning underlying meaning requires semantic comprehension—the exact strength of modern language models.
18.5 Calling a language model from Python
Python provides standard libraries for interacting with language models:
Online API services: Official SDKs (openai, anthropic, google-genai) offer top-tier reasoning capabilities. Online APIs require an API key and incur per-token charges. Keys should always be stored in environment variables (e.g., via a .env file), never committed directly into Git repositories.
Local models via Ollama: The free, open-source Ollama platform runs open models (such as Google Gemma, Meta Llama, or Mistral) locally on your machine. The Python ollama package communicates with the local runner. Local models require zero API keys, run completely free of charge, and guarantee that participant data never leaves your computer.
18.5.1 Structured outputs with Pydantic
Qualitative coding requires two properties that standard web chats lack:
System Prompt: A fixed, reproducible codebook defining the seven themes and explicit tie-breaking rules.
Structured Enforcement: Guaranteeing that the model’s response is strictly restricted to one of the seven valid theme names, avoiding verbose conversational preambles.
In Python, structured outputs are defined using Pydantic:
from pydantic import BaseModelfrom typing import Literalclass QualitativeCoding(BaseModel): theme: Literal["Supervision", "Workload", "Finances", "Family", "Health", "Isolation", "Other" ]# The qualitative codebook prompt:codebook ="""You are helping an academic researcher code open-ended survey answers from graduatestudents, who were asked: 'What has been your biggest challenge during your studies?'Assign each answer to exactly ONE theme: the single primary challenge the student describes.If multiple challenges are mentioned, select the one the student presents first or emphasizes most.Themes:- Supervision: supervisor guidance, feedback, meetings, availability, or disagreements.- Workload: too much work or too little time; deadlines, thesis writing, coursework, readings.- Finances: tuition fees, living expenses, funding, scholarships, debt, paid employment.- Family: children, spouse, caring responsibilities, family commitments, home life.- Health: physical illness, mental health, stress, anxiety, exhaustion, burnout, insomnia.- Isolation: loneliness, social alienation, lack of peer network, studying remotely.- Other: administrative issues, research ethics approvals, statistical software, language barriers."""
Using Ollama in Python, a classification query is executed with zero sampling temperature (temperature=0) to ensure deterministic repeatability:
import ollamaresponse = ollama.chat( model="gemma4:e4b", messages=[ {"role": "system", "content": codebook +"\nReply with the theme name only."}, {"role": "user", "content": "I cannot afford the tuition fees and rent this term."} ], options={"temperature": 0})print(response['message']['content'].strip())# Output: Finances
For the graduate wellbeing study, all 536 student answers were coded using a local instance of Google Gemma 4 (gemma4:e4b) via Ollama. The run script (data-raw/run_ai_coding.py) logged the model specifications, execution timestamp, and coded each validation response twice to test stability.
18.6 Evaluating AI qualitative coding
The model’s classifications are loaded from the archived results file and merged with Elaf’s 200 hand-coded responses:
Model: gemma4:e4b (Ollama (local))
Validation sample: 200 students
AI Coding Accuracy: 90.0%
AI Cohen's kappa: 0.874
The language model achieves 90.0% accuracy and a Cohen’s kappa of 0.874, representing almost perfect agreement(Landis and Koch 1977). This substantially outperforms the keyword dictionary (61.0% accuracy, \(\kappa = 0.512\)).
The diagonal cells represent cases of complete agreement. The off-diagonal cells represent disagreements. Examining the specific discordant text entries provides valuable qualitative insight:
When inspectable, disagreements frequently occur on multi-clause answers where two genuine challenges are present (e.g., struggling with workload while dealing with health fatigue). Inspecting these discrepancies often leads to refining the codebook definitions rather than pointing to model failure.
18.6.2 What the students reported (RQ11)
With the model’s reliability rigorously established (\(\kappa = 0.874\)), the automated coding is applied across all 536 student responses (Figure 18.1):
theme_shares = ( ai_results['ai_theme'] .value_counts(normalize=True) .sort_values(ascending=True) *100)fig, ax = plt.subplots(figsize=(6.5, 3.5))ax.barh(theme_shares.index, theme_shares.values, color="#2f6793", edgecolor="none", height=0.65)ax.set_xlabel("Share of answers (%)")ax.grid(True, linestyle="--", alpha=0.5, axis="x")for i, v inenumerate(theme_shares.values): ax.text(v +0.8, i, f"{v:.1f}%", va="center", fontsize=9.5)ax.set_xlim(0, max(theme_shares.values) +6)plt.tight_layout()plt.show()
Figure 18.1: Primary challenge reported by graduate students, coded by a local language model.
Supervision (33.0%) and Workload (22.8%) represent the two dominant challenges encountered by graduate students.
18.6.3 Validating against quantitative survey scales
To establish construct validity, we cross-reference the qualitative AI themes with students’ quantitative ratings of supervisor support from the Chapter 2 questionnaire:
The qualitative text and quantitative survey data validate each other: students whose primary open-ended challenge was classified as Supervision report significantly lower perceived supervisor support (2.82) than students reporting any other challenge (3.30 – 3.46).
18.7 Using AI responsibly in academic research
18.7.1 Verify and maintain authorship accountability
AI tools cannot be listed as authors on academic publications (as confirmed by COPE and major journal publishers) because an AI model cannot take legal or intellectual responsibility for scholarly claims. You remain solely responsible for the accuracy of every finding, citation, and line of code in your thesis.
18.7.2 Protect participant data and ethical boundaries
Before transmitting text to cloud-hosted AI APIs, verify that your institutional review board (IRB) ethics approval and participant consent forms explicitly authorize external data sharing. When dealing with sensitive, non-consented, or human-subject data:
Strip all direct and indirect identifying details (names, institutions, specific dates).
Utilize local offline models (e.g., via Ollama), ensuring zero bytes of text leave your research machine.
18.7.3 Maintain computational reproducibility
LLMs are updated continuously by commercial vendors. To ensure reproducible research:
Record the exact model identifier (e.g., gemma4:e4b), host provider, temperature (0), and date.
Archive the exact system prompts and codebook files.
Save the model’s generated outputs as permanent data artifacts (.csv), analyzing the frozen outputs rather than querying dynamic endpoints repeatedly.
18.7.4 Transparent disclosure
Disclose AI assistance transparently in your thesis methodology or acknowledgments:
“Open-ended challenge responses were qualitatively categorized into seven themes using a local open-weight language model (Gemma 4, e4b variant, via Ollama and Python). Agreement with the primary author’s manual coding of a random benchmark sample (N = 200) was validated using Cohen’s kappa (\(\kappa = 0.874\)).”
18.8 Common misconceptions
“High percent agreement proves that qualitative coding is accurate.” High raw agreement often reflects category skew; Cohen’s kappa is essential to evaluate agreement above chance expectations.
“The AI knows the factual answer.” Language models generate plausible linguistic patterns, not verified facts; AI classifications must be validated against human ground truth.
“Model responses are permanently identical.” Cloud models change across releases and non-zero temperatures introduce stochastic variation; outputs must be archived for reproducibility.
“Using AI in a thesis is forbidden.” Most academic institutions welcome transparent, disclosed, and validated AI assistance; what is forbidden is unverified, undisclosed, or fabricated content.
18.9 Chapter review
18.9.1 Summary
Inter-rater reliability measures agreement between independent coders, corrected for chance using Cohen’s kappa (\(\kappa\)).
Large language models predict sequential tokens based on training patterns; while proficient in Python data workflows, they are prone to confident hallucinations and lack access to external data unless explicitly provided.
Effective AI assistance requires specific prompts detailing goals, data structures, and target libraries, combined with rigorous manual cross-checking.
Keyword dictionaries offer a transparent baseline for text classification but fail on complex syntactic context.
Python interacts with LLMs via official APIs or local Ollama instances; Pydantic ensures reliable structured enum outputs.
Local LLMs can match human qualitative coding accuracy (\(\kappa > 0.80\)) while preserving participant privacy.
Responsible AI usage demands human accountability, data protection, deterministic parameters (temperature=0), reproducibility archives, and explicit disclosure.
18.9.2 Key terms
Qualitative coding, codebook, inter-rater reliability, Cohen’s kappa, large language model, token, hallucination, prompt engineering, system prompt, temperature, local model, Ollama, Pydantic, structured output, construct validity, reproducibility, disclosure.
18.10 Exercises
The interactive Playground provides browser-based exercises with solutions:
Request an AI assistant to write a pandas aggregation computing average sleep hours by faculty and semester. Test the code on a tiny 3-row DataFrame containing a missing value, and verify that missing data is handled appropriately.
Expand the keyword dictionary with three additional regular expressions, and measure whether Cohen’s kappa improves on the 200-student test set.
Inspect 10 instances where the keyword dictionary disagrees with Elaf’s hand coding. Determine whether the discrepancy stems from keyword ambiguity, negation, or multiple mentioned challenges.
Classify a sample of 20 open-ended responses using a local language model, and compute the agreement rate with the human benchmark.
In the two-coder toy example, modify Coder B so that it assigns 100% of responses to “Workload”. Observe the resulting raw agreement and Cohen’s kappa.
18.11 Further reading
Gilardi et al. (2023), “ChatGPT outperforms crowd-workers for text-annotation tasks”, PNAS, examining LLM zero-shot classification versus human raters.
Cohen, J. (1960), “A coefficient of agreement for nominal scales”, Educational and Psychological Measurement.
Landis, J. R., & Koch, G. G. (1977), “The measurement of observer agreement for categorical data”, Biometrics.
The official Ollama documentation at ollama.com for running open-weight models locally.
References
Cohen, Jacob. 1960. “A Coefficient of Agreement for Nominal Scales.”Educational and Psychological Measurement 20 (1): 37–46. https://doi.org/10.1177/001316446002000104.
Landis, J. Richard, and Gary G. Koch. 1977. “The Measurement of Observer Agreement for Categorical Data.”Biometrics 33 (1): 159–74. https://doi.org/10.2307/2529310.
