Chapter 7 — Reps

Every rep moves your study forward. By Friday, a classmate should be able to take your design doc and rerun your plan without asking you one question. That is the bar.

Ground rules

• Work on your real research project — the domain you chose in Week 2 and its applied-AI question. No toy topics.
• Everything lands in your portfolio repo: design doc, configs/experiment-design.yaml, the baseline matrix, the seeded harness. Commit as you go; the commits are your pre-registration.
• When you use an LLM to help draft or critique, log the model name + version + date, keep the human-in-the-loop, and verify every fact and citation it gives you (Appendix C). The model accelerates; you decide and you are accountable.
• “I’ll define that later” is the sound of a hole opening. Fill it now.

Rep 1 — Fill the design spine (no empty fields)

Copy code/experiment-design.yaml into your repo at configs/experiment-design.yaml and fill every field for your study: RQ, hypothesis + null, baselines, conditions, ablations, data + version, metrics + direction, seeds, significance test, environment.

mkdir -p configs && cp code/experiment-design.yaml configs/experiment-design.yaml
$EDITOR configs/experiment-design.yaml
git add configs/experiment-design.yaml && git commit -m "Lock experiment design spine"

Reflect (3–4 sentences): Which field was hardest to fill, and why? An empty field is an unanswered research question — name the one you’re least sure of.

Rep 2 — State the claim and its falsifiable hypothesis

Write your primary claim in one sentence, then the hypothesis (H1) and null (H0) with a concrete threshold and an alpha. “RAG is better” is not falsifiable; “RAG raises execution accuracy by ≥ 3 points absolute, alpha = 0.05” is.

Reflect: What exact result would falsify your hypothesis? If you can’t name it, the hypothesis isn’t sharp enough yet — fix it.

Rep 3 — Name a naive AND a strong baseline (and pledge to tune it)

List your floor baseline (the dumbest thing that could work) and your strong baseline (current accepted practice — usually the thing AI is meant to replace). Write one sentence on how you’ll tune the strong baseline fairly so you’re not beating a strawman.

Reflect: Be honest — were you tempted to pick a baseline you knew you’d beat? Why is a win over a sandbagged baseline a loss in review? (Prov 11:1, ESV.)

Rep 4 — One ablation per claim

Open code/baseline-matrix.csv, replace the example rows with your study, and ensure every claim has at least one ablation that removes a single component to isolate the effect.

cp code/baseline-matrix.csv configs/baseline-matrix.csv
$EDITOR configs/baseline-matrix.csv     # one row per claim x condition

Reflect: For your main claim, which single ablation would most embarrass you if it showed your gain came from “more tokens,” not your idea? Run that one.

Rep 5 — Pick the benchmark and confront contamination

Choose an established benchmark for your task (or justify a custom eval set). If your method is an LLM and the benchmark is old/public, write a two-sentence contamination threat and decide whether to switch to a contamination-resistant / live benchmark (e.g., LiveCodeBench, MMLU-Pro).

Reflect: Does a high score on your benchmark measure capability or memorization? Tie this back to construct validity from Chapter 6.

Rep 6 — Plant a leakage bug, then kill it

In a throwaway notebook, build a tiny classifier two ways: (a) fit_transform a scaler on the whole dataset then split; (b) split first, fit on train only, transform test. Compare the accuracies.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
# (a) LEAKY: scale before split  -> inflated
# (b) CLEAN: split, fit on train, transform test  -> honest

Reflect: How big was the gap? Kapoor & Narayanan (2023) found leakage across 294 papers in 17 fields — explain in two sentences why this bug is so easy to ship without noticing.

Rep 7 — Define the split and the leakage guard

Write your train/val/test policy into the YAML: which split, fixed before any tuning, test touched once. State the rule “preprocessing fit on train only,” and if you’ll cross-validate, note that preprocessing goes inside the CV pipeline (StratifiedKFold for classifiers).

Reflect: When, exactly, are you allowed to look at the test set? Write the one sentence that governs it.

Rep 8 — Make the multi-seed harness yours

Run code/multiseed_eval.py as-is, then replace run_one_seed with your real train+eval (or a faithful stand-in if compute isn’t ready), keeping the seed discipline and the planned test.

python code/multiseed_eval.py

Reflect: Report your mean ± std across 5 seeds. How wide is the spread? If a single run could land anywhere in that band, what does that say about anyone’s single-run “win”?

Rep 9 — Choose the significance test and effect size in advance

Decide now how you’ll test your difference: ASO (deep-significance, τ = 0.2) for multi-seed DL scores, or scipy.stats.permutation_test for a small benchmark. Name the effect-size measure you’ll report alongside the p-value. Write both into the YAML.

Reflect: Why is committing to the test before the run a matter of integrity, not just tidiness? (Connect to HARKing / p-hacking — Chapter 12.)

Rep 10 — Walk the reproducibility checklist

Open code/repro_checklist.txt, tick every box you can honestly tick for your design as it stands, and compute your score (boxes checked / total).

Reflect: List your three lowest-confidence boxes and the concrete fix for each. If you’re below ~0.8, your study isn’t reproducible yet — say what you’ll change this week.

Rep 11 — Run the Experiment Design Checker twice

Use the on-page Experiment Design Checker widget. Answer honestly for your study now; record the score and every missing piece it names. Fix the holes in your YAML, then run it again.

Reflect: What was the gap between your first and second score? Which single fix moved it most?

Rep 12 — Pin your LLM instrument (if you have one)

If any part of your method or pipeline calls an LLM, add a block to the YAML recording the model name, version, date, temperature, and how you’ll log them per run. If no LLM, write the one-line reason your pipeline is deterministic without that note.

Reflect: A closed model can change weights between June and July. In one sentence, how does an un-pinned model break reproducibility?

Done? One Last Thing.

Assemble Reps 1–12 into the first full draft of your Experimental Design Document (the P7 deliverable): claim + falsifiable hypothesis, baselines (naive + strong), ablations (one per claim), benchmark + contamination note, splits + leakage guard, primary metric + direction, ≥ 5 seeds, the pre-chosen significance test + effect size, and the reproducibility plan (env, seeds, data version, code commit, LLM pinning, disclosure).

Then do the real test: hand the document and your repo to a classmate. Have them read it and tell you, out loud, exactly what they’d run — every command, every split, every seed — without asking you anything. Every question they have to ask is a hole. Write down each hole, fix it, and commit. That commit is your pre-registration: the design is locked.

Up next: Project 7