Chapter 11 — Full Experimental Execution

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman, Cargo Cult Science (1974)

“You shall not bear false witness against your neighbor.” — Exodus 20:16 (ESV)

Why This Matters

For ten weeks you have been getting ready to be wrong. That is what all of it was for. The literature review, the comparison matrix, the sharpened question, the falsifiable hypothesis, the reproducible design, the pinned environment, and last week’s pilot — every one of those was scaffolding around a single moment that arrives this week: you run the real experiment and find out what is actually true. Not what you hoped. Not what would make a clean story. What is true.

This is the week the practicum stops being about planning research and becomes research. The pilot proved the pipeline runs end to end on a small slice. Now you scale it up — full dataset, all your seeds, every baseline, every ablation your design document promised — and you collect the primary results that Chapter 12 will analyze and Chapter 14 will write up. The deliverable is not a polished figure or a confident claim. It is something humbler and far more important: the raw results dataset with full provenance — every number, tied to the exact seed, config, code commit, and (if an LLM was your instrument) the model version and date that produced it. This is worth 15% of your grade, and it is graded on integrity and completeness, not on whether your method won.

Read that grading criterion twice, because it inverts the instinct you have carried through every prior course. You are used to being graded on the quality of the answer. This week you are graded on the honesty of the record. A practicum is the first half of a thesis, and a thesis is not a sales pitch for a method — it is a faithful account of an investigation. Plenty of important papers report that the obvious approach didn’t work; what makes them publishable is not the disappointment but the rigor and honesty with which the disappointment was established. You are practicing that posture now, on a small scale, where the stakes are a grade rather than a career.

That last sentence is the whole chapter. Because here is the temptation, and it is real, and it will visit you personally this week: a run will come back ugly. Your method will lose to the baseline on seed 3. A configuration you were sure about will produce numbers that embarrass your hypothesis. And a quiet voice will suggest that seed 3 was “probably a fluke,” that the bad config “wasn’t really the one we meant,” that you could just… not write that row down. That voice is the subject of this chapter. The discipline that answers it is called provenance, and the federal government has a three-word name for what happens when you give in: fabrication, falsification, plagiarism — the FFP triad (42 CFR Part 93).

The AI thread runs straight through the middle of this. If your domain uses a closed, hosted model — a GPT, a Claude, a Gemini — as an instrument in your experiment, you have a reproducibility problem the previous generation of researchers never had: your instrument can change underneath you without warning. The model you called “GPT-5.5” in June is not guaranteed to be byte-identical to the one you call by the same name in August. So provenance for AI experiments means logging the exact prompt, the temperature, the model version string, and the date — because “we asked the model” is not a recordable measurement, and a result you cannot tie to a versioned instrument is a result you cannot defend.

Which brings us to the question underneath the week: what does it mean to bear true witness to your data? The eighth commandment — “You shall not bear false witness against your neighbor” (Exodus 20:16, ESV) — is usually heard as a courtroom rule, and it is. But Luther’s explanation in the Small Catechism widens it: we are not only to refrain from lying about our neighbor but to “defend him, speak well of him, and explain everything in the kindest way.” A dataset is a kind of neighbor — a witness whose testimony you are responsible for relaying faithfully to everyone who will ever read your paper. To drop the run that didn’t work is to put words in that witness’s mouth. To report only seed 7 because it looked best is to bear false witness. This week you learn the craft that makes honest reporting possible: a ledger so complete that lying to yourself becomes hard and lying to your reader becomes obvious.

Let’s run it.

11.1 — What “Full Execution” Actually Means

The pilot answered one question: does the pipeline run? Full execution answers a different one: what does the pipeline say when you run everything you promised? The gap between those is bigger than it looks, and it is where most students lose a week they did not budget for.

Your experimental design document (Chapter 7) and your proposal (Chapter 8) made a set of concrete promises. Full execution is the act of keeping every one of them and recording what came back. Before you launch anything, turn those promises into a checklist — the experimental matrix.

Coach’s Note — The number-one schedule killer this week is discovering, at launch, that your “full” run is 40× the compute of your pilot and your free Colab session times out at hour 12. You should already know your full-run budget from the pilot — time per cell × number of cells × number of seeds. If you don’t, compute it before you launch anything. A run you cannot finish is worse than a run you never started, because it eats the week and leaves you with a half-populated, untrustworthy ledger.

Count the cells before you launch. If your design says “method + 1 baseline + 1 ablation, across 8 seeds, on 1 test split,” that is 3 × 8 × 1 = 24 runs — and your final log should contain exactly 24 result rows plus any documented reruns. Knowing the target row count is the cheapest integrity check you own: when the runs finish, you either have 24 rows or you have a question to answer, and “a question to answer” is infinitely better than a silent gap you discover during paper-writing.

The deliverable is the raw record. Not analyzed, not aggregated, not pretty. Chapter 12 does statistics; Chapter 13 does meaning. This week your only job is to produce numbers you can stand behind — and the standing-behind is entirely a matter of provenance.

11.2 — Provenance: The Four Things Every Number Needs

A result without provenance is a rumor. Here is the rule, and it is not negotiable: every number in your raw dataset must be reproducible from what you logged. That means, for every single result row, you can answer four questions without guessing.

1. What code produced it? → the Git commit hash (git rev-parse HEAD), and clean — no uncommitted changes. A result from a dirty working tree is a result you cannot reproduce, because you no longer know what code ran.
2. What configuration produced it? → the full config (hyperparameters, paths, flags) — ideally a versioned config file, not flags you typed and forgot.
3. What randomness produced it? → the seed, set everywhere (Python random, NumPy, your framework). Determinism was Chapter 9’s discipline; this is where it pays off.
4. What data produced it? → the dataset version/hash and the exact split, so a reader knows you didn’t quietly swap the test set.

For a deterministic, code-only experiment, those four are enough. The starter file code/run_log_template.csv gives you one row per run with exactly these columns, and code/provenance.py shows the harness side. Wire your experiment to append a row automatically — never by hand, because the hand is where the lies and the typos both live. The single most valuable habit you can build this week is that recording a result is part of producing it: the run is not “done” until its provenance row exists. Make the logging a step the experiment cannot skip, and the integrity takes care of itself.

Here is the minimum machine-written record at the start of every run:

import subprocess, json, datetime, hashlib

def provenance(seed: int, config: dict, dataset_path: str) -> dict:
    head = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip()
    dirty = bool(subprocess.check_output(["git", "status", "--porcelain"]).strip())
    with open(dataset_path, "rb") as f:
        data_hash = hashlib.sha256(f.read()).hexdigest()[:16]
    return {
        "run_id": datetime.datetime.now().strftime("%Y%m%dT%H%M%S"),
        "commit": head,
        "dirty": dirty,            # if True, your result is NOT reproducible
        "seed": seed,
        "config": json.dumps(config, sort_keys=True),
        "dataset_sha256_16": data_hash,
        "timestamp_utc": datetime.datetime.utcnow().isoformat(),
    }

If dirty is ever True in your final dataset, you have a hole in your evidence. Commit first, then run. Always.

Why the dataset hash, and not just the path? Because a path is a promise about a file, and files move, get re-downloaded, get re-split, and get silently corrupted. The SHA-256 is the file itself, fingerprinted — it is how a reader (and future-you) confirms that the “test set” in run 1 is byte-identical to the “test set” in run 40, and that neither was quietly swapped or re-shuffled mid-experiment. Leakage between splits (Kapoor & Narayanan 2023 found it across 294 papers in 17 fields) often hides exactly here, in an unnoticed re-split. A hash per run is a cheap, permanent witness that your data stayed put.

11.3 — When the LLM Is the Instrument: AI-Specific Provenance

If your applied-AI project calls a hosted model as part of the experiment — an LLM agent doing code review (SE), an LLM phishing detector (security), an LLM-as-SQL-optimizer (databases), an LLM grading student submissions (edtech) — then the model is a piece of lab equipment, and lab equipment needs a calibration record. The hard problem, as of mid-2026, is that closed hosted models drift. The endpoint behind a name can be updated; sampling is stochastic; a provider can deprecate a version mid-study. A result you cannot pin to a versioned instrument is not reproducible.

So your provenance grows four AI-specific columns, captured per call, not per project:

Coach’s Note — Set temperature to 0 for any experiment where you want the model’s behavior measured rather than its variance — and then still log it and still run multiple seeds, because temperature 0 is not a guarantee of determinism on hosted endpoints (batching, hardware, and silent updates leak nondeterminism in). If your variable of interest is the model’s run-to-run spread, raise the temperature deliberately and report the distribution, not a cherry-picked sample. Either way: log the setting. A measurement whose instrument settings are unknown is not a measurement.

Make it concrete with a domain example. Say your project evaluates an LLM agent for code review (the SE thread — think SWE-bench Verified’s 500 human-validated tasks, or the c-CRAB code-review-agent benchmark, arXiv:2603.23448). Your experimental “result” for a task is whether the agent’s patch makes the test suite pass. That looks deterministic — tests pass or they don’t — but the path to it is not: the agent’s reasoning is sampled, its tool calls vary, and the hosted model behind your API name can shift between Tuesday’s runs and Thursday’s. So your provenance row for each task carries the commit of your harness, the seed of your sampler, and the model version string + date + verbatim prompt of the instrument. Six weeks later, when a reviewer asks “would this reproduce?”, the honest answer for a closed model is “to within the model provider’s stability over that window” — and the only way to say even that much is to have logged the window.

There is also a sharper integrity trap here. A 2026 study on long-context Q&A (arXiv:2603.08274) found that LLM fabrication rises with context length — the model invents more as you feed it more. If your instrument is a model and your data are its outputs, you are measuring a thing that can confidently make facts up. That is fine as the object of study — “how often does the model hallucinate?” is a great RQ — but it is fatal if you let the model’s fluent-but-false output silently become a row in your own dataset without verification. The model can fabricate. You may not.

The disclosure rule travels with this: whichever venue you are targeting, AI use as part of the method gets disclosed per its policy. NeurIPS 2026 wants non-standard LLM method use described in the experimental-setup section; ACM and IEEE want it named in the Acknowledgements with the system identified. See Appendix C for the venue-by-venue map. Log now so disclosing later is bookkeeping, not archaeology.

11.4 — Recording What You Saw, Not What You Hoped

Here is the discipline that separates a researcher from a salesperson: you record observations as they happen, before you know whether they help you. The lab notebook — physical or digital — is written in the present tense, in the moment, and is never edited after the fact to look better.

Keep a run log (the narrative) alongside the results dataset (the numbers). The dataset has the metrics; the log has what the metrics couldn’t capture: the run that OOM’d at hour 6, the config you fat-fingered, the moment you noticed the validation curve looked wrong, the surprise that you don’t yet understand. The starter code/run_log_template.csv and the narrative pattern in code/experiment_journal.txt give you both halves.

Three rules for the moment of recording:

• Append, never overwrite. A result, once written, is evidence. If you rerun, write a new row with a new run_id — don’t replace the old one. The history of what you tried is part of the truth.
• Write the anomaly down immediately, even when — especially when — you don’t understand it yet. “Seed 3 method underperforms baseline by 4 points, unclear why, flagged for analysis” is a sentence that protects you. The future you who explains it (or honestly reports it as unexplained) will thank the present you who recorded it.
• No silent deletion. If a run is genuinely invalid — the job crashed, the wrong dataset loaded, a bug you have identified — you may exclude it, but you record that you excluded it and exactly why, with the evidence. An exclusion you can defend in writing is science. An exclusion you do quietly is falsification.

Here is the discipline in one concrete picture. Suppose at hour six the validation curve for one condition flattens early and you don’t know why. The wrong move is to say nothing and hope it resolves. The right move is a single timestamped line in the journal: “seed 3, val loss plateaus at epoch 9, expected to keep dropping — cause unknown, flagged for analysis.” You have now done three things at once: you have protected the future-you who must explain it, you have made it impossible to quietly pretend you didn’t notice, and you have created a thread Chapter 13 can pick up honestly — whether the ending is “it was a learning-rate issue” or “we report it as an open anomaly.” An unexplained result, honestly recorded, is science. An unexplained result, silently smoothed over, is the first step off the cliff.

Coach’s Note — Falsification is not only inventing data (that’s fabrication). The federal definition (42 CFR Part 93) is broader: falsification includes “manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented.” Read that word again: omitting. Dropping the inconvenient seed is, by the literal regulatory definition, falsification. The honest path costs you a worse-looking number this week and buys you a defensible paper forever.

11.5 — The Temptation to Drop the Runs That “Didn’t Work”

Let’s name the specific failure, because it is the one that catches good people. You have eight seeds. Seven support your hypothesis. One — seed 3 — is ugly. Every fiber wants to call seed 3 an outlier and move on. Here is how to think about it honestly.

A run is legitimately excludable only when you can point to a mechanical failure independent of the result: the process crashed, the GPU threw an error, the wrong file loaded, an identified bug ran. In those cases the run did not measure your method — it measured a broken pipeline — and you exclude it with documentation and rerun.

A run is not excludable because the number is disappointing. “Seed 3 is an outlier” is a hypothesis about the data, and you do not get to assert it to make a graph cleaner. If seed 3 ran the same code, config, and data as the others and merely produced a result you dislike, it is a real measurement of your method’s variance — which is exactly the thing Bouthillier et al. (2021) showed you must report. The spread is the finding. A method that wins on 7 of 8 seeds and loses on 1 is a more honest and more useful result than a method that “wins” because you deleted the loss.

There is a subtler cousin of seed-dropping, and it is more dangerous because it feels like diligence: rerunning until it behaves. You don’t delete seed 3 — you just rerun the whole experiment “to be safe,” notice the second batch looks better, and quietly keep the second batch. This is the garden of forking paths. Each individual rerun is defensible; the selection among them is not. The fix is the same fix as everywhere in this chapter: log every run, keep them all, and decide in advance how many runs you will collect and report. A decision made before you see the data cannot be corrupted by the data.

And notice what the disappointing run actually gives you. A method that wins on 7 of 8 seeds is telling you something true and useful — that it is usually better but not reliably better — and that is a finding a reviewer will respect and a future researcher can build on. The cherry-picked “wins on 8 of 8” is a lie that will, sooner or later, fail to reproduce in someone else’s hands, and your name will be on it. The honest spread is the gift; the clean lie is the liability.

The Data Integrity Ledger below is built to make this fork visible: it will flag a missing seed in your sequence and ask you, by name, where seed 3 went.

11.6 — Provenance Infrastructure: From CSV to Experiment Trackers

A CSV you append to by hand works for a tens-of-runs project and is the right place to start. Past that, the field has tooling. As of mid-2026, the common reproducibility chain looks like this:

• In-experiment logging — write provenance into your results from inside the code (the snippet in §11.2). Don’t trust memory; trust the machine that ran it.
• Experiment tracking — tools that auto-capture metrics, params, code version, and environment per run, so you don’t reconstruct it later. (Choose one and use it consistently; the point is automatic capture, not the brand.)
• Environment capture — you pinned this in Chapter 9: pip freeze / conda env export, transitive locks via conda-lock, system libs in Docker. The frozen environment is part of every result’s provenance.
• Archival + a citable identity — when the dataset is final, deposit it to Zenodo (CERN/OpenAIRE), which mints a DOI per deposit and, since its Oct 2024 integration, auto-deposits public source code to Software Heritage with a SWHID linked to the DOI. Tag a release in Git, deposit it, and your raw dataset has a permanent, citable address.

A note on where the raw record lives. Your run log is the index; the per-run artifacts (model checkpoints, full prediction files, per-example outputs) are the body. Keep them together and addressable — a results directory named by run_id, with the log row pointing at it — so that “row 17 of the dataset” resolves to an actual folder a reader can open. The discipline is simple: nothing in the log should be unreachable, and nothing in the results directory should be unrecorded in the log. When the two halves agree, your dataset is whole.

This is also the moment ACM’s Artifact Review & Badging vocabulary starts to matter (badges: Artifacts Available, Artifacts Evaluated, Results Validated — where Reproduced means an independent team got your result using your artifacts and Replicated means without them). You are building toward “Available” and “Reproduced” right now, this week, by logging provenance you’d otherwise have to reconstruct under deadline. See Appendix A for the environment on-ramp and Appendix B for the toolkit.

Coach’s Note — “I’ll clean up the provenance after the runs finish” is the lie this chapter exists to kill. You will not. The runs will finish at 3 a.m., the deadline will be Friday, and the seed-to-commit mapping you didn’t capture will be gone. Build the logging before you launch. Five minutes of git rev-parse HEAD in your harness saves the week.

11.7 — Orchestrating the Full Run Without Losing Your Week

The full matrix is many runs, often long ones, frequently on borrowed compute that will cut you off. Three orchestration disciplines turn that from a gamble into a plan.

Make every run resumable. Checkpoint state periodically and resume from the last checkpoint on restart. A free Colab or Codespaces session that dies at hour 12 should cost you minutes, not the whole cell. A run you cannot resume is a run any timeout can destroy — and a destroyed run is a hole in your dataset.

import os
ckpt = "results/ckpt_seed{seed}_{cond}.pt"
if os.path.exists(ckpt):
    state = load(ckpt); start = state["step"]      # resume
else:
    start = 0                                       # fresh
for step in range(start, total_steps):
    train_one_step(...)
    if step % 100 == 0:
        save({"step": step + 1, "model": ..., "rng": rng_state()}, ckpt)

Note the rng in the checkpoint: to keep a resumed run reproducible, you must restore the random state, not just the weights. Otherwise the second half of your run uses different randomness than a clean run would, and your provenance quietly lies.

Drive the matrix with a script, not your fingers. A loop over conditions and seeds (the pattern in the exercises) means the matrix is defined once, in code you can read and rerun, instead of in a shell history you’ll never reconstruct. The script is documentation of what you ran.

Fail loud, fail logged. When a cell errors, your harness should write a row with status=excluded and the reason — before it moves on. The worst outcome is a run that silently didn’t happen and leaves no trace, because three days later you cannot tell the difference between “seed 6 failed” and “I forgot seed 6.” Both look like a gap; only one is honest, and the log is what tells them apart.

Coach’s Note — The asymmetry in that table is the lesson. A recorded failure is cheap — you see it, you rerun it, you move on. An unrecorded absence is expensive, because it is invisible until the worst possible moment, and it is indistinguishable from negligence even when it was just bad luck. Spend your engineering effort making failures loud and gaps impossible. A pipeline that fails noisily and logs everything is worth more than one that’s fast and quiet.

11.8 — Human-Subjects and Log Data: A Quick Integrity Gate

If your domain touches people — an HCI study, AI-generated feedback graded by students, telemetry or interaction logs, an edtech evaluation — you have an ethics gate to pass before you collect, not after. Two anchors, as of mid-2026:

• The Belmont Report’s three principles: respect for persons, beneficence, justice.
• The Common Rule (45 CFR 46), with eight exempt categories at 46.104(d). Critically: exemption is determined by your IRB or institutional official, not self-certified. You do not get to decide your own study is exempt.

Venue practice varies — CHI 2026, for instance, defers to “the appropriate ethics review requirements that apply to the authors’ research environment” rather than mandating a US IRB. The rule of thumb: if you are collecting data from humans, ask your institution’s review office this week, before the run, whether you need review. A brilliant result built on data you weren’t cleared to collect is unpublishable and worse. (Anonymized data generally leaves GDPR scope; pseudonymized data usually does not — and the CJEU’s EDPS v. SRB ruling, Sept 2025, makes that classification depend on the recipient. When in doubt, ask.)

11.x — Interactive Lab: Data Integrity Ledger

Below this chapter on the page is the Data Integrity Ledger — your hands-on rehearsal for the deliverable. It is the run log made interactive and, crucially, adversarial: it is built to catch the corners you’d be tempted to cut.

Log each run as you “execute” it: enter the seed, config, commit, model version (if AI is your instrument), and result. As you fill the ledger, the tool does three things you should let it teach you:

1. It flags missing provenance. Leave the commit blank, or log a run from a dirty tree, and the ledger marks the row unreproducible — a visible hole in your evidence.
2. It detects a dropped run. Log seeds 1, 2, 4, 5 and the ledger asks, pointedly, where is seed 3? — the exact temptation from §11.5, surfaced before it becomes a habit.
3. It scores your record. A complete, gap-free, fully-provenanced ledger earns a “defensible” score; a tidy-but-incomplete one does not. The lesson is that looking clean and being honest are different things, and only one of them survives review.

Spend fifteen minutes logging a realistic run sequence — including a crash you exclude with a reason, and a disappointing seed you keep. Then carry the same discipline into Project 11. The widget is practice; the project is the rep that counts.

11.9 — Bearing True Witness to Your Data

We come back to the commandment. “You shall not bear false witness against your neighbor” (Exodus 20:16, ESV). It is striking that of the ten words God gives at Sinai, one is reserved for testimony — for the integrity of what we say happened. Truthful witness is not a peripheral virtue God tacks on after the big ones; it sits alongside “you shall not murder” and “you shall not steal,” because a community runs on trust in testimony, and false testimony quietly destroys what violence destroys loudly.

Science is a community that runs entirely on trust in testimony. You will never personally rerun most of the experiments you cite; you trust the witness of the researchers who ran them. When you publish your raw dataset, you join that chain of witnesses, and a stranger five years from now will build on your numbers without re-deriving them. To fabricate, to falsify, to quietly drop the inconvenient seed — that is to corrupt the chain at the link entrusted to you. It is, precisely, false witness against a neighbor you will never meet.

And here is the part that the secular framing of “research integrity” can name but not fully explain: why be honest when the bad number costs you and no one would catch the deletion? Feynman’s answer — “you are the easiest person to fool” — is true and good, but it is finally pragmatic: lie and you’ll fool yourself into worthless work. The Christian answer goes deeper. You work coram Deo — before the face of God — who sees seed 3 whether or not your reviewer does. The eighth commandment is not a rule you keep to avoid getting caught; it is a description of who you are before a God who is himself the Truth (John 14:6). Luther’s catechism widens “do not bear false witness” into the positive duty to “explain everything in the kindest way” — and your data, that voiceless neighbor, deserves exactly that: to be explained kindly and truthfully, every row of it, the ugly ones included.

It is worth saying plainly that this cuts against an instinct the academy quietly trains into you. The pressure is all toward the positive result — the method that wins, the clean graph, the publishable story — and a whole literature on publication bias and the file-drawer problem testifies to how that pressure bends honest people. The eighth commandment is a wall against exactly that current. It does not ask whether your result is impressive; it asks whether your report is true. Those are different questions, and a great deal of bad science lives in the gap between them. The Christian researcher answers the second question first and lets the first fall where it may, because his standing does not rest on the result. It rests on Christ, which means he can afford to lose an experiment without losing himself — and that security is precisely what makes integrity affordable when the bad number lands.

There is a freedom in this you will feel by Friday. The researcher who has decided, in advance, to report whatever comes back is free in the lab in a way the cherry-picker never is. He does not have to manage his data, spin his runs, or remember which version of the story he told. He runs the experiment, writes down what happened, and rests. “The truth will set you free” (John 8:32, ESV) is, among other things, an extraordinarily good description of what a clean, complete, honest ledger does for a researcher’s sleep.

11.10 — Common Pitfalls

Pitfall: Logging provenance by hand, after the fact. Example: You finish 40 runs, then sit down to fill in the seed-and-commit spreadsheet from memory and shell history. Fix: Write provenance from inside the harness at run start (§11.2). The machine that ran the experiment is the only honest witness to what it ran.

Pitfall: Running from a dirty working tree. Example: You tweak a hyperparameter, don’t commit, launch the run; the result is now untraceable to any version of the code. Fix: Commit before every run. Make your harness record git status --porcelain and refuse — or at least loudly flag — a dirty launch.

Pitfall: Dropping the disappointing seed. Example: Seed 3 underperforms; you call it an outlier and report the other seven. Fix: Exclude only on a documented mechanical failure (§11.5). A merely bad number is a measurement of your method’s variance — keep it; the statistics in Chapter 12 are built to handle it.

Pitfall: Treating a hosted LLM as a fixed instrument. Example: You log “we used GPT-5.5” and nothing else; three weeks later the endpoint has shifted and your numbers won’t reproduce. Fix: Log the exact model version string, the date, the verbatim prompt, and the sampling params per call (§11.3). A drifting instrument needs a timestamped calibration record.

Pitfall: Letting the model’s output become your data without verification. Example: Your LLM instrument fabricates a fact in a long-context run and that fabricated value lands, unchecked, as a row in your dataset. Fix: Verify model outputs against ground truth before they enter the record. The model may fabricate; your dataset may not (§11.3).

Pitfall: A “full” run you can’t actually afford to finish. Example: Your full matrix is 40× the pilot; the free Colab session dies at hour 12 and you have a half-populated, untrustworthy dataset. Fix: Compute the full-run budget from the pilot before launching (§11.1). Stage long runs with checkpointing so a timeout costs you minutes, not the week.

Pitfall: Overwriting results instead of appending. Example: A rerun replaces the old row; the history of what you tried — and what changed — is silently lost. Fix: Append-only logging with a fresh run_id per run (§11.4). The record of every attempt is part of the truth, not clutter to be cleaned up.

Pitfall: Rerunning until the numbers behave, then keeping the good batch. Example: You rerun the experiment “to be safe,” the second batch looks better, and you quietly report the second batch. Fix: Keep every batch and decide the run count in advance (§11.5). The forking-paths selection is the dishonesty, even when each individual run is clean.

Pitfall: Collecting human or log data before the ethics gate. Example: You start logging user interactions for an HCI study, then learn afterward you needed institutional review. Fix: Ask your review office before the run whether the study needs review (§11.8). Exemption is determined by the IRB, never self-certified — and unauthorized data is unpublishable.

11.11 — Reps

The reps for this week are in the exercises, and they are not toy problems — every one moves your research project forward toward the deliverable. A preview:

• Wire automatic provenance into your experiment harness so every result row carries commit, seed, config, and dataset hash with no human typing.
• Launch the full matrix — all seeds, baselines, and ablations your design promised — with a budget you computed in advance.
• Keep the experiment journal in the present tense, recording at least one anomaly the moment you notice it.
• Handle a real exclusion honestly — document one mechanical failure you excluded and one disappointing result you kept.
• Run the AI-instrument calibration rep (if applicable) — log version, date, prompt, and temperature per call, and verify outputs against ground truth.

Then test yourself on the Check Your Reps quiz on this page before you move on — five questions on provenance, the exclusion fork, and the FFP triad.

11.12 — This Week’s Deliverable

Your deliverable is the Raw Results Dataset — full spec in Project 11, worth 15% of your course grade (the “Experimental Results” weight in the deliverable schedule). It is the complete primary record of your experiment: every metric from every cell of your design matrix, each row tied to its full provenance (seed, config, commit, dataset hash, and — if AI is your instrument — model version, date, prompt, and sampling params), plus a narrative run log documenting anomalies, any excluded runs with their reasons, and the environment you ran in.

It is graded on integrity and completeness, not on whether your method won. A complete, honest dataset where your method underperforms is a strong submission. A clean-looking dataset with a dropped seed is a failing one. The starter files in code/ — the run-log template, the experiment journal pattern, the provenance snippet, and the repro checklist — are your skeleton.

Next week, in Chapter 12, you analyze this dataset: significance tests, effect sizes, confidence intervals, and the statistical honesty that turns raw numbers into a defensible Results section. Everything Chapter 12 can claim depends on the integrity of what you collect this week.

11.13 — Coach’s Final Word

This is the week you find out whether you are a scientist. Not because the experiment is hard to run — you’ve been ready for that since Chapter 7. Because somewhere in the next seven days a number is going to come back wrong, and no one will be watching when you decide what to do with it.

Write it down. The ugly seed, the embarrassing config, the result that makes your hypothesis look weaker than you sold it in Week 8 — write all of it down, with its provenance, in a ledger so complete you couldn’t lie if you wanted to. You are not collecting evidence for the conclusion you want. You are bearing witness to what is true. That is a smaller, harder, and far better thing, and it is the only kind of research that lasts.

The researcher who reports whatever comes back sleeps fine. Be that one.

See you on Monday.

Up next: the exercises for the reps · Project 11 for the Raw Results Dataset deliverable · then Chapter 12. Previous: Chapter 10. Reference: Appendix A (research environment), Appendix B (researcher’s toolkit), Appendix C (using AI responsibly), Appendix D (glossary).