Playground: Chapter 18
Using AI with Python
This page practises the ideas of Chapter 18 in four parts. Practise the chapter works through the book’s exercises, with starter code, hints, and solutions. Go further goes beyond the book, with new questions about agreement and about checking code. Check your understanding asks short questions whose answers open with a click. Do it yourself gives open tasks with no answers: you decide how to use and check AI in your own work, as you will in your thesis.
Calling a language model needs a local model on your own computer (Ollama) or an API key, so that step is practised in the download project. Everything around it runs here in Python: checking code an assistant suggests, the keyword method, Cohen’s kappa, and the language model’s saved results from the chapter, which the book’s script produced once with a local model (Gemma 4, e4b version, through Ollama). In the exercises, replace each ______ with your own code and press Run Code.
Practise the chapter
These are the exercises at the end of Chapter 18, with the same numbers. In the setup, open_responses_coded holds the 200 hand-coded answers (theme), ai_coding the language model’s saved theme for every answer, keywords and code_by_keywords() the chapter’s dictionary method, and cohen_kappa_score() calculates Cohen’s kappa.
Exercise 1: Checking an assistant’s code
An assistant, asked for the average sleep in each faculty and semester, suggested the code below. Run it, check it with a tiny example and by another route, and fix it.
The numpy function that computes the arithmetic mean while ignoring NaNs is np.nanmean. In pandas, .mean() ignores NaNs by default (skipna=True).
np.nanmean([6, 7, np.nan])In pandas, .groupby().mean() automatically skips NaN values, yielding 6.64 hours for Education in semester 1. However, if working in NumPy or custom code without skipna=True, a single np.nan makes the entire result nan. Using np.nanmean() ensures missing values are dropped. The check by another route—summing values and dividing by the non-missing count—gives the exact same 6.64. Always verify assistant code against edge cases like missing values.
Exercise 2: Three more keywords
Add three keywords to the dictionary that you think would fix some of its mistakes, and report whether the kappa improves. Discuss whether you are now fitting the dictionary to these answers.
Add a word of your choice inside the Workload pattern, for example course. Any word you add must appear in the answers to change anything.
new_keywords["Workload"] = r"\b(time|deadline|workload|too much|assignment|reading|busy|thesis|work|course)"The original dictionary agrees with the hand coding on 61% of the answers (kappa 0.51). With thesis, work, and course added to Workload, and isolat and on my own added to Isolation, the agreement rises only to 61.5%, and the kappa does not improve (0.51): every new word fixes some answers and breaks others (“work” also appears in answers about jobs and money). Adjusting the dictionary while looking at these 200 answers is the overfitting of Chapter 11: a fair test needs an independent validation sample.
Exercise 3: Reading the disagreements
Read ten answers where the keyword coding disagrees with the hand coding, and say whether you would agree with the hand coding in every case.
Filter for rows where theme != keyword_theme.
check["theme"] != check["keyword_theme"]The keywords disagree with the hand coding on 78 of the 200 answers. In most cases, the hand coding is clearly right, and the keyword failure is instructive: subtle phrasing like “the comments I get are very brief” relates to supervision without mentioning “supervisor” or “meeting”. Reading disagreements reveals where rule-based regex falls short and where language models excel.
Exercise 4: The language model’s coding
Look at what the chapter’s language model did with the same 200 answers: its saved themes are in ai_coding.
Kappa compares the hand coding with the model’s theme in model_check['ai_theme'].
cohen_kappa_score(model_check['theme'], model_check['ai_theme'])The model agrees with the hand coding on 18 of the first 20 answers (90%), and on 90.0% of all 200 answers, with a Cohen’s kappa of 0.874 (almost perfect agreement, compared to 0.510 for regex keywords). Re-running the model at temperature 0 yielded 100% identical outputs (repeatability = 1.000).
Exercise 5: A methods paragraph for AI coding
Write the paragraph for your own methods section describing how you would use and validate AI coding in a study of your own. Write your answer first, then open the model answer.
“Open-ended responses were coded into seven themes defined in a codebook (Appendix X). A random sample of 200 responses was first hand-coded by the researcher. All responses were then coded by an open-weight large language model (Gemma 4 e4b, executed locally via Ollama, temperature 0.0) using structured JSON output via Pydantic. Agreement between the model and the researcher on the hand-coded sample was evaluated with Cohen’s kappa, and disagreements were qualitatively reviewed; automated coding was adopted given substantial inter-coder agreement (κ = 0.874). Stability was confirmed by re-evaluating the sample under identical parameters (100% replication). No confidential data was transmitted to external servers. System prompts, codebook schemas, and raw inference tables are archived in the supplementary materials.”
Exercise 6: A coder who always says Workload
In the two-coder example at the start of the chapter, change the second coder so that it puts every answer under Workload. Calculate the raw agreement and kappa, and explain the result.
The second coder assigns "Workload" to all 20 responses.
coder_b = ["Workload"] * 20Raw agreement is 80% (16 out of 20), yet Cohen’s kappa is exactly 0.00. The second coder did no actual classification; it agreed on the 16 Workload responses simply by assigning the majority class to every case. Cohen’s kappa subtracts expected chance agreement, exposing that zero discriminative skill was applied.
Go further
These exercises go beyond the book.
Exercise 7: Kappa with three themes
Two coders read 20 answers and agree on 16. Calculate agreement and kappa as the majority class (“Workload”) expands from 10 to 80 responses while the four errors remain constant.
Compare c_a with c_b.
kap = cohen_kappa_score(c_a, c_b)With 10 Workload answers, raw agreement is 0.800 and kappa is 0.680. As Workload expands to 80 answers, raw agreement reaches 0.956, but kappa rises only to 0.781. The four errors persist, but because the majority class inflates chance agreement, raw agreement flatters the coders. Kappa keeps the performance on rarer classes in perspective.
Exercise 8: The dictionary’s most common confusion
Find the most common misclassifications made by the keyword dictionary compared to hand coding.
Grouping by ['theme', 'keyword_theme'] counts occurrences of each specific confusion pair.
errors = check[check["theme"] != check["keyword_theme"]]The most frequent confusion is Workload answers coded as Other (10 cases): answers describing workload without matching regex patterns fall through to the fallback. Next are Isolation answers misclassified as Supervision (8) or Other (6). Isolation is the most challenging theme for simple regex dictionaries.
Exercise 9: Long answers and short answers
Count the words in each hand-coded answer, and compare the average length by theme and accuracy.
str.split().str.len() counts whitespace-delimited words in pandas.
check["words"] = check["biggest_challenge"].str.split().str.len()Supervision responses are the longest on average (~21 words) while Isolation responses are the shortest (~15 words). Furthermore, misclassified responses are longer on average (20 words) than correctly classified ones (17 words): longer answers often mention multiple secondary challenges, misleading simple keyword tallies.
Exercise 10: Three mistakes an assistant might make
Examine these common pitfalls in AI-generated data analysis code:
Notice that students['faculty'] == 'education' yields 0 because categories are capitalized (“Education”). Also ensure grouped proportions normalize within groups rather than across the whole dataframe.
# 1. Age mean: 29.7 years
# 2. Faculty: 148 students in Education
# 3. Grouped percentage: normalizes within each facultyThese illustrate three critical classes of AI coding errors: silent handling of missing values, string case sensitivity mismatches, and incorrect denominator aggregation (within-group vs global rates).
Check your understanding
Answer each question in your own words first, then click to see a model answer.
1. Why can raw agreement between two coders be misleading?
Because some agreement occurs purely by chance. When one theme is disproportionately frequent, two coders can agree frequently even if they are guessing. Cohen’s kappa subtracts expected chance agreement, ensuring that a coder who simply assigns the majority class to every case receives a score of 0.
2. What is a hallucination?
A fluent, confident, but factually false statement generated by a language model—such as a fabricated function name, invalid argument, non-existent citation, or incorrect statistical formula. Hallucinations must be detected through code execution, testing against synthetic toy data, and reading official documentation.
3. Why is a temperature of 0 used when a model codes research data?
Temperature controls the randomness of token generation. Setting temperature to 0 forces greedy decoding (selecting the highest probability token), ensuring deterministic, consistent, and reproducible qualitative coding across runs.
4. What must be recorded so that AI coding can be reproduced and checked?
The model provider, exact model identifier and checkpoint hash, execution date, temperature and sampling settings, full prompt templates, structured schema definitions, software versions, and the raw text output log.
5. When must data not be sent to an online model?
Whenever responses contain personally identifiable information (PII), confidential participant disclosures, or sensitive institutional records, unless explicit institutional review board (IRB) approval, participant consent, and data processing agreements permit third-party cloud hosting. A local model (e.g., via Ollama) completely avoids sending data over the network.
Do it yourself
These tasks have no starter code and no answers. The code space is empty and ready to run; data and evaluation tools are preloaded.
1. Invent a new open question, such as “What would most improve your supervision?”, and write ten plausible answers of your own. Write a codebook of four or five themes, code the answers yourself, calculate Cohen’s kappa against a peer’s or model’s coding, and inspect disagreements.
2. In the download project, modify a theme definition in the codebook, evaluate 20 responses with a local model, and observe how kappa changes.
3. Draft the disclosure paragraph for using AI assistance in your own thesis, specifying tools, tasks, verification steps, and supplementary archives.
Work on your own computer
The project contains the data, the language model’s saved results, and all four parts of this page as a Python script (exercises.py), with complete answers in solutions.py. It also includes guidance for running local models through Ollama or online models with your own API key (see README.txt).