Chapter 7 — Designing a Reproducible Experiment

“An experiment is a question which science poses to Nature, and a measurement is the recording of Nature’s answer.” — Max Planck

“…having followed all things closely for some time past, to write an orderly account for you… that you may have certainty concerning the things you have been taught.” — Luke 1:3–4 (ESV)

Why This Matters

Last week you made a claim you could be wrong about. You wrote a falsifiable hypothesis, named your independent and dependent variables, and drew up a threats-to-validity table. Good. But a hypothesis is a promise, and this week you have to design the machine that keeps it — or breaks it.

Here is the uncomfortable number. In 2016, Nature surveyed 1,576 scientists (Baker, Nature 533:452–454). More than 70% had tried and failed to reproduce another scientist’s experiment. More than half had failed to reproduce their own. That survey was cross-disciplinary — treat it as framing, not a CS measurement — but computer science is not exempt. We have an extra hazard the wet-lab sciences don’t: our experiments are code, and code that “works on my machine” is the oldest lie in the field. Add a frontier model whose weights change under you between June and July, and reproducibility stops being a virtue and becomes the whole ballgame.

A reproducible experiment is not a nicer version of an experiment. It is the only kind that counts as evidence. If another researcher cannot walk your path — same data, same splits, same seeds, same environment, same commands — and arrive at numbers close enough to yours that the conclusion survives, then you have produced a story, not a result. Reviewers know this. The reflex question on every empirical CS submission is now: could I rerun this? This week you make the answer yes, on paper, before you’ve burned a single GPU-hour.

This is also where the applied-AI thread gets sharp. Your domain — whatever you chose in Week 2 — has a question that AI might answer better than the old way: RAG vs a fine-tuned model for SQL generation, an RL controller vs a hand-tuned congestion algorithm, an LLM phishing detector vs a classic filter. The temptation in 2026 is to run your method once, see a number bigger than the baseline’s, and call it a win. That instinct is the enemy. Deep-learning results swing wildly with the random seed alone (Bouthillier et al., MLSys 2021). A single run that “beats the baseline” is statistically empty. Your design has to plan for that variance before you collect a single datapoint.

And here is the apologetic question Luke hands us. When Luke opens his Gospel, he does not say trust me. He says he followed all things closely and wrote an orderly account so that Theophilus may have certainty. That is a reproducibility statement two thousand years early. The point of an orderly account is that someone else can check it. Why must another person be able to walk the same path you walked? Hold that question — we’ll come back to it. For now: design like a stranger is going to grade you, because one will.

Coach’s Note — Most of this course’s “do the work” weeks live in the back half. This week is different. You don’t run anything yet. You write the plan so airtight that running it becomes bookkeeping. The single biggest predictor of whether your results survive peer review is whether you locked the design before you saw the data.

7.1 — The shape of an empirical CS experiment

Strip away the domain and almost every empirical CS study has the same skeleton: a claim, a method you think is better, one or more baselines it has to beat, a dataset with honest splits, a metric that operationalizes “better,” and a protocol that says how many times you’ll run it and how you’ll decide you won. Your job this week is to fill every one of those slots, in writing, with no holes.

The hole is the danger. An undefined split is where leakage hides. An unnamed baseline is where a reviewer says “compared to what?” A single run is where luck masquerades as a finding. Design is the discipline of closing holes before they cost you a month.

We’ve put a machine-readable skeleton in code/experiment-design.yaml. It is not decoration — it is the spine of this week’s deliverable. Open it now. Every field you can’t fill is a question you haven’t answered yet.

study:
  research_question: "RQ1: Does RAG produce more executable SQL than a LoRA baseline?"
  hypothesis: "H1: RAG raises execution accuracy by >= 3 points absolute (alpha=0.05)."
baselines: [zero-shot prompt, LoRA-tuned]
conditions: [RAG top-k=5]
ablations: ["k=1", "retrieval off / random context"]
metrics: {primary: execution accuracy, direction: higher-is-better}
protocol: {seeds: [0,1,2,3,4], significance_test: "ASO (tau=0.2)"}

Coach’s Note — Write the YAML before the prose. Prose lets you hand-wave; a config field is either filled or empty. The empty fields are your to-do list.

7.2 — Baselines: better than what?

A result has no meaning in isolation. “Our model scored 74.5%” tells a reviewer nothing. 74.5% versus what? A baseline is the comparison that gives your number meaning, and you need at least two kinds.

The naive baseline is the floor — the dumbest thing that could work. Zero-shot prompting with no retrieval and no tuning. Majority-class prediction. The cache policy that evicts at random. If your fancy method can’t beat the floor, stop. The floor protects you from fooling yourself.

The strong baseline is the number a reviewer will actually hold you to: the current accepted approach in your domain. For an applied-AI study this is usually the thing AI is supposed to replace — the hand-tuned heuristic, the classic filter, the fine-tuned model you’re claiming RAG beats. Beating the floor is table stakes. Beating the strong baseline is the contribution.

The integrity hazard here is specific and common: the weak-baseline win. It is easy to make your method look good by comparing it to a baseline you didn’t bother to tune. That is not a lie, exactly — but it is a false balance (Prov 11:1, ESV: “A false balance is an abomination to the LORD, but a just weight is his delight”). Tune your baseline as hard as you tune your method. A win over a strawman is a loss in review. We’ve laid out the baseline/ablation plan in code/baseline-matrix.csv — one row per claim per condition, with the role each plays.

7.3 — Ablations: which part actually carries the effect?

You beat the strong baseline. Wonderful. Why? If your method has three new parts — a retriever, a reranker, and a bigger prompt — and you only report the all-on number, you’ve shown the bundle works but explained nothing. Worse, a reviewer can’t tell whether the gain came from retrieval or just from feeding the model more tokens.

An ablation removes exactly one component and re-measures. One thing at a time. If accuracy collapses when you turn the retriever off, retrieval is carrying the effect. If it barely moves when you swap real retrieved context for random context, then — uncomfortable truth — your gain was just more tokens, not retrieval, and you’d better know that before a reviewer finds it.

Full method:        RAG, top-k=5, real retrieval     -> 74.5%
Ablation A (k):     RAG, top-k=1                      -> 72.9%   (k helps a little)
Ablation B (retr.): RAG, top-k=5, RANDOM context     -> 71.4%   (retrieval is the story)
Strong baseline:    LoRA, no retrieval               -> 71.0%

The rule of thumb: one ablation per claim. If you claim “retrieval improves executability,” you owe an ablation that turns retrieval off. The ablation is how you earn the word because.

Coach’s Note — Ablations feel like extra work that can only make your method look worse. That fear is exactly why they’re trustworthy. A paper that ablates its own contribution is a paper a reviewer believes.

The same pattern transfers to every domain on the menu, because the shape of the design doesn’t change — only the nouns do. A few worked translations of “claim → baselines → ablation”:

Notice the constant: the strong baseline is almost always the established, non-AI (or simpler-AI) approach your method is supposed to displace. That is the comparison the contribution lives or dies on. Pick yours now, and pick it honestly.

7.4 — Benchmarks and the contamination problem

A benchmark is a shared dataset-plus-metric the field has agreed to measure on, so your number is comparable to everyone else’s: Spider for text-to-SQL, SWE-bench Verified for code agents (500 human-validated GitHub tasks where success = the test suite passes), and so on. Use an established benchmark when one exists — inventing your own makes you incomparable and slightly suspicious.

But benchmarks carry a 2026-specific hazard when your method is an LLM: test-set contamination. If the benchmark’s questions and answers were on the open web, they were probably in the model’s training data, and a “high score” may be memorization, not capability. This is the live form of the construct validity question from Chapter 6 — does a high score measure reasoning, or recall?

The field’s partial defenses, as of mid-2026:

• n-gram decontamination — flag and remove training examples that overlap the test set. GPT-3 used 13-gram overlaps; GPT-4 raised the bar to 40-grams. It is imperfect: a paraphrase slips right past it.
• Contamination-resistant / “live” benchmarks — LiveCodeBench, LiveBench, MMLU-Pro, FrontierMath continuously refresh items so models can’t have memorized them.

Be careful with the published contamination numbers — they are contested and frequently mis-paired across sources. (The “~29% of MMLU contaminated” and “22.9% inflation removed on GSM8K” figures, for instance, come from different papers and measure different things; pairing them is a common mistake.) If you cite one, cite it to its paper, point-in-time, and hedge. For your design this week, the action item is concrete: if your method is an LLM and your benchmark is old and public, say so as a threat to validity, and prefer a live or freshly-built evaluation set if you can.

Coach’s Note — Contamination is the construct-validity threat dressed up in 2026 clothes. A memorized answer and a reasoned answer produce the same benchmark score, which means the score has quietly stopped measuring what you claimed. If you can’t rule contamination out, you don’t delete the result — you disclose it as a limitation and stop calling it “reasoning.”

7.5 — Splits and the leakage trap

Train, validation, test. The most basic discipline in empirical ML, and still where studies die. The rule: you train on train, tune (hyperparameters, model selection, “let me try one more thing”) on validation, and you touch test exactly once, at the very end, to report. Peeking at test to make decisions is how you publish a number that won’t replicate.

Data leakage is when information from outside the training set sneaks into training. Kapoor & Narayanan (2023, Patterns 4(9):100804) found leakage across 294 papers in 17 fields — this is not a beginner’s mistake, it is an epidemic. The classic bug is scaling or feature-selecting on the whole dataset before you split:

# WRONG — leakage. The scaler has seen the test set's distribution.
X = scaler.fit_transform(X_all)
X_train, X_test = split(X)

# RIGHT — fit on train only, then transform the rest.
X_train, X_test = split(X_all)
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)        # transform, never fit

For classifiers, use scikit-learn’s cross_val_score / cross_validate with StratifiedKFold — it’s the default for estimators that inherit ClassifierMixin, and it preserves class balance across folds. (scikit-learn 1.9.0 as of 2026-06; verify the version live.)

from sklearn.model_selection import cross_val_score, StratifiedKFold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(pipe, X_train, y_train, cv=cv)   # pipe = preprocessing INSIDE the fold

Note the comment: put preprocessing inside the cross-validation pipeline so it refits per fold. That one habit kills the most common leakage bug.

Coach’s Note — A leakage bug doesn’t crash. It hands you a beautiful, too-good number that feels like success. The grief comes later, when an honest rerun deflates it. Design the split now, in writing, and the temptation never gets a foothold.

7.6 — Metrics: operationalizing “better”

You picked a metric in Chapter 6. Now make it precise enough to compute the same way twice. “Accuracy” is not precise — accuracy of what, measured how? For text-to-SQL, execution accuracy (does the predicted query return the gold result set?) and exact-match accuracy (does the SQL string match?) can disagree by a lot — a query can be worded differently and still be right. Name the exact one, name its direction (higher-is-better for accuracy, lower-is-better for latency), and write down the function that computes it.

Two more cautions. First, on imbalanced data, accuracy lies — a phishing detector that flags nothing scores 99% accuracy if 99% of mail is legitimate, while catching zero attacks. Report precision and recall (and say which one your task actually cares about: missing an attack vs annoying a user). Second, for any latency or cost metric, report a percentile (p50, p95), not just the mean — one pathological slow query can hide behind a pretty average.

A study can have one primary metric and a few secondary ones — but decide before you run which is primary. Choosing the metric that happened to win after you see the data is a form of the p-hacking we’ll dismantle in Chapter 12. Lock it now.

7.7 — How many runs, and how you’ll decide you won

Here is the discipline that separates a finding from a fluke. Run more than once. Deep-learning scores move under the random seed alone — different initialization, different data shuffle, different result (Bouthillier et al., MLSys 2021; arXiv:2103.03098; see also Picard’s tongue-in-cheek “torch.manual_seed(3407) is all you need,” arXiv:2109.08203). A single run is a single sample from a noisy distribution.

So: at least 5 seeds, report mean ± std, and never report a bare single number for a learned system. Then pick your significance test in advance:

• For multi-seed deep-learning score vectors, the field-appropriate tool is the Almost Stochastic Order (ASO) test from the deep-significance package (v1.2.5; arXiv:2204.06815). It makes weaker distributional assumptions than a t-test — the recommended acceptance threshold is τ = 0.2.
• For a non-parametric comparison on a small benchmark, scipy.stats.permutation_test.
• And always report an effect size, not just a p-value. A statistically “significant” gain of 0.1% is practically nothing.

The harness in code/multiseed_eval.py shows the structure: seed everything, run each condition five times, report mean ± std, and run the planned test. It’s a template — swap in your real pipeline, keep the seed discipline.

   baseline: mean=0.7106  std=0.0118  n=5
     method: mean=0.7450  std=0.0121  n=5
absolute diff = +0.0344   Welch t = 4.41   p = 0.0023   Cohen's d = 2.88

Coach’s Note — Pre-register the test in your design doc. If you run the experiment, then go shopping for the test that gives you p < 0.05, you have stopped doing science and started doing marketing. Decide once, before the data.

7.8 — Reproducibility from the design: env, seeds, data, code

The four pillars of “a stranger can rerun this.” You won’t build the environment until Chapter 9, but you design for it now, because retrofitting reproducibility is misery:

• Environment — you will pin dependencies (pip freeze, conda env export), lock transitive deps (conda-lock), and containerize system libraries (CUDA, gcc, ffmpeg) with Docker when they matter. Write down the hardware (e.g., 1× A100 40GB).
• Seeds — seed torch/numpy/random; set torch.use_deterministic_algorithms(True) and torch.backends.cudnn.benchmark=False; on CUDA ≥ 10.2 set CUBLAS_WORKSPACE_CONFIG=:4096:8. (Determinism still isn’t guaranteed across releases or CPU-vs-GPU — note that as a limitation.)
• Data — pin the dataset version or DOI. “We used Spider” is not enough; “Spider 1.0, this commit / this dataset-card URL” is.
• Code — a Git repo, a pinned commit, and a README whose commands reproduce each table. At submission you’ll archive a tagged release to Zenodo, which mints a DOI and (since the Oct 2024 integration) auto-deposits your source to Software Heritage with a SWHID linked to the DOI.

A taste of what “pinned” looks like in practice — none of this is hard, but all of it has to be decided now and done consistently later:

# Environment capture (Chapter 9 makes this a discipline; design for it here)
conda env export --no-builds > environment.yml      # exact package versions
pip freeze > requirements.txt                        # pip-installed deps
# Determinism (PyTorch), set at the top of every run script:
#   torch.manual_seed(seed); np.random.seed(seed); random.seed(seed)
#   torch.use_deterministic_algorithms(True)
#   torch.backends.cudnn.benchmark = False
export CUBLAS_WORKSPACE_CONFIG=:4096:8               # CUDA >= 10.2, else it raises
git tag v1.0-design && git rev-parse HEAD            # the commit your numbers tie to

One more, specific to applied AI: if an LLM is your instrument, log its name, version, date, and temperature. Closed models drift. “GPT-5.5” in April and “GPT-5.5” in July may not be the same weights — and as of mid-2026 flagship models turn over on the scale of weeks, not years. A result you can’t tie to a specific model version on a specific date is a result no one — including future-you — can reproduce.

The full set of boxes is in code/repro_checklist.txt. Tick them as you write the design doc — anything below ~0.8 means your study is not yet reproducible.

7.9 — Pre-registration: locking the plan before the data

The strongest move available to you this week is to pre-register: write the design down, time-stamp it read-only, before you collect data. It is the structural cure for HARKing (Hypothesizing After the Results are Known — Kerr, 1998) and for p-hacking (Simmons, Nelson & Simonsohn, 2011, Psychological Science).

Two free tools, as of 2026:

• OSF (Open Science Framework, by the Center for Open Science) — create a time-stamped, read-only registration of your plan before data collection. COS also promotes Registered Reports, where a venue accepts the plan in principle before results exist.
• AsPredicted.org — a lighter-weight, nine-question pre-registration form.

You don’t have to pre-register publicly for the practicum. But you will lock your design doc with a Git commit this week, and treat that commit the way a pre-registration treats its timestamp: the plan is fixed, and any later change is a documented amendment, not a quiet edit. That single habit is most of research integrity in practice.

7.10 — Design as the answer to Chapter 6’s validity threats

Last week’s threats-to-validity table wasn’t busywork — it was the list of ways your study could be wrong, and this week’s design is how you defend against each one. The four categories (Cook & Campbell, 1979; adapted for software engineering by Wohlin et al.) map cleanly onto the design choices you just made:

If you can point at a design decision for every row, your study is defensible. If a row has no corresponding choice, that’s the threat that will surface in review — go back and add the control, the ablation, or the disclosure now. A design document is, at bottom, a written argument that you’ve thought about how you could be wrong and built guards against it.

7.x — Interactive Lab: Experiment Design Checker

Below this chapter on the website is the Experiment Design Checker — use it now, with your own study in mind.

It walks you through the design questions this chapter raised: Do you have a naive baseline? A strong one? At least one ablation per claim? A named primary metric with a direction? Defined train/val/test splits with a leakage guard? At least five seeds? A significance test chosen in advance? A pinned environment, data version, and code commit? As you answer, it computes a reproducibility-and-validity score and — more usefully — names the missing pieces: the unnamed baseline, the claim with no ablation, the single-run protocol.

Run it twice. First, answer honestly for your study as it stands right now; note the score and the holes. Then fix the holes in code/experiment-design.yaml and run it again. The gap between the two scores is the work this chapter is asking of you. Don’t argue with the checker — it’s standing in for a reviewer, and the reviewer is less forgiving.

7.y — Why must another walk the same path?

Return to Luke. He had, by his own account, access to eyewitnesses and to accounts already in circulation. He could have simply asserted. Instead he tells Theophilus how he knows: he followed all things closely, in order, that you may have certainty (Luke 1:3–4, ESV). The reliability of the account is offered as something checkable, not something to be taken on the author’s charisma.

That is the theological root of reproducibility, and it cuts against a deep temptation in research: the desire to be believed rather than checked. We want our result to stand on our cleverness. Scripture and good science agree that it should stand on something a stranger can verify. “The one who states his case first seems right, until the other comes and examines him” (Prov 18:17, ESV) — which is exactly what peer review is. An experiment another person cannot rerun is a case stated first with no one allowed to examine it.

There is humility built into this. To design for reproduction is to admit you might be wrong, to hand a stranger everything they’d need to prove it, and to want them to try. That is not weakness; it is the posture of someone searching out the truth rather than defending a position. “It is the glory of God to conceal things, but the glory of kings is to search things out” (Prov 25:2, ESV) — and a thing genuinely searched out can be searched out again, by anyone, and found the same. The reproducible experiment is an act of honesty toward your neighbor: you are making it possible for them to know what you know, instead of asking them to trust that you know it. When you pin a seed, document a split, and disclose your AI use, you are, in a small and concrete way, loving the next researcher as yourself.

7.z — Common Pitfalls

Pitfall: The single-run win. Example: You run your RAG method once, get 74.5% to the baseline’s 71%, and write “our method outperforms the baseline.” Fix: Five seeds minimum, report mean ± std, run a pre-chosen significance test, report effect size. One run is one sample from a noisy distribution (Bouthillier et al. 2021).

Pitfall: The strawman baseline. Example: You compare your tuned method to an untuned, default-hyperparameter baseline and declare victory. Fix: Tune the baseline as hard as the method. A win over a baseline you sandbagged is a false balance — and a desk-reject when a reviewer reruns it properly.

Pitfall: Leakage before the split. Example: You fit_transform a scaler or run feature selection on the whole dataset, then split into train/test. Fix: Split first; fit preprocessing on train only; transform (never fit) the test set. Put preprocessing inside the CV pipeline so it refits per fold.

Pitfall: Touching the test set to make decisions. Example: You check test accuracy, tweak a hyperparameter, check again, repeat. Fix: Tune on validation. Touch test exactly once, at the end, to report. Every peek inflates the number and kills replication.

Pitfall: Choosing the metric (or test) after seeing the data. Example: F1 didn’t win, so you report accuracy instead; the t-test wasn’t significant, so you switch to a one-tailed test. Fix: Declare the primary metric and the significance test in the design doc, before any run. Lock it with a commit. Later changes are documented amendments, not silent swaps.

Pitfall: Benchmark contamination treated as a capability. Example: Your LLM scores high on an old, public benchmark and you report it as reasoning ability. Fix: Note contamination as a construct-validity threat; prefer a contamination-resistant / live benchmark (LiveCodeBench, MMLU-Pro) or a freshly built eval set; cite any contamination figure to its specific paper.

Pitfall: The un-pinned LLM instrument. Example: Your method calls “the API”; six weeks later the model has changed and your numbers won’t reproduce. Fix: Log the model name, version, date, and temperature. A closed model that drifts is a reproducibility threat — design around it, disclose it.

7.(z+1) — Reps

The full set lives in the exercises. They move your study forward this week — by Friday you should be able to hand someone your design doc and have them rerun your plan without asking you a single question. A preview:

• Rep 1 — Fill every field of code/experiment-design.yaml for your study; no field left empty.
• Rep 3 — Name your naive and strong baselines and write one sentence on how you’ll tune the strong one fairly.
• Rep 4 — Write one ablation per claim into code/baseline-matrix.csv.
• Rep 6 — Plant a leakage bug in a toy pipeline, watch the inflated number, then fix the split and watch it deflate.
• Rep 8 — Run code/multiseed_eval.py over 5 seeds and replace the stub with your real metric.

Then do the on-page Check Your Reps quiz below the chapter — five questions straight from this material.

7.(z+2) — This Week’s Deliverable

This week you produce the Experimental Design Document (P7) — the artifact a second researcher could pick up and rerun. Full spec in Project 7. It is the pre-execution lock for everything in the back half of the course: get the splits, baselines, ablations, metrics, seeds, and reproducibility plan right now, and Weeks 9–12 become execution instead of improvisation. It feeds directly into next week’s proposal, where you defend this design in front of your peers.

For environment setup, see Appendix A; for the literature and template toolkit, Appendix B; for the AI-disclosure and integrity rules you must honor, Appendix C; for any term that’s new, Appendix D.

7.(z+3) — Coach’s Final Word

A reproducible experiment is a promise you make to a stranger: here is everything you need to check me. It is the least glamorous week in this course and the most load-bearing. The flashy weeks — the results, the figures, the talk — all rest on the design you lock this week. Get it right and you’ll execute with calm in Week 11 while your classmates are debugging splits and re-running for seeds they forgot to set.

Design like a reviewer is going to rerun you. Tune your baselines like you want to lose. Pick your metric before the data can tempt you. Pin everything. And remember why: not to be believed, but to be checkable — to write an orderly account, so that the next person may have certainty.

Lock the design. See you on Monday.

Up next: the exercises for the reps · Project 7 for the Experimental Design Document · then Chapter 8, where you defend this design under fire. Previous: Chapter 6.