Chapter 11 — Reps

The keyboard is the gym, and this week the gym is your own experiment. Every rep below moves your project toward the deliverable — the Raw Results Dataset with full provenance. No toy problems. By Friday you should have run the real thing and recorded it honestly.

Ground rules

• Work on your own project. Use the domain and design you’ve carried since Week 1–2. These reps assume you have a working pipeline (Chapter 9) and a passing pilot (Chapter 10).
• Commit before you run. Always. A result from a dirty tree (§11.2) is unreproducible.
• Append, never overwrite. Every run is a new row with a new run_id (§11.4).
• Record before you judge. Write the number down before you decide whether you like it.
• AI policy for the reps: AI may help you write logging and harness code and explain errors. AI may not generate, edit, or “clean up” your results — that’s the data. If an LLM is your experimental instrument, log it per §11.3 and verify its outputs.
• Starter files: code/provenance.py, code/run_log_template.csv, code/experiment_journal.txt, code/repro_checklist.txt.

Rep 1 — Wire automatic provenance into your harness

Adapt code/provenance.py to your project. At the top of every run, capture commit, seed, config, and dataset hash from inside the code and append a row.

git rev-parse HEAD        # this must be CLEAN — git status --porcelain should be empty
python -m your_experiment --seed 1 --condition method   # writes results/run_log.csv
head -2 results/run_log.csv

Reflect (3–4 sentences): Where in your old workflow were you trusting your memory for provenance? What’s the one field you were most likely to lose if you logged by hand later?

Rep 2 — Compute your full-run budget before launching

From your pilot’s wall-clock-per-cell, compute: time/cell × #conditions × #seeds. Write the total, and your compute ceiling (Colab/Codespaces session limit, cluster quota, or local hours).

Reflect: Does your full run fit your budget? If not, what gives — fewer seeds (and what that costs you per Bouthillier et al. 2021), checkpointing, or a smaller matrix? Decide now, in writing, not at hour 12.

Rep 3 — Add checkpointing so a timeout costs minutes, not the week

Make long runs resumable: save state periodically, resume from the last checkpoint on restart.

# sketch — adapt to your framework
if os.path.exists(ckpt): state = load(ckpt); start = state["step"]
else: start = 0
for step in range(start, total):
    ...
    if step % 100 == 0: save({"step": step, ...}, ckpt)

Reflect: Simulate a kill (Ctrl-C mid-run) and resume. Did you lose any results? Any provenance? A run you can’t resume is a run a timeout can destroy.

Rep 4 — Launch the full matrix

Run every cell your design document promised: all seeds × all conditions (method, baselines, ablations) × splits. Let the harness append a provenance row per run.

for cond in method baseline ablation_no_attention; do
  for seed in 1 2 3 4 5 6 7 8; do
    python -m your_experiment --seed "$seed" --condition "$cond"
  done
done

Reflect: How many rows landed in your log? Does that equal #conditions × #seeds (minus documented exclusions)? Any mismatch is a hole — find it before you call the run done.

Rep 5 — Keep the experiment journal in the present tense

Using code/experiment_journal.txt, log at least one anomaly the moment you notice it — even if you don’t understand it yet.

Reflect: Write the anomaly entry here verbatim. Then answer: would you have remembered this on Friday if you hadn’t written it down now? (The honest answer is almost always no.)

Rep 6 — Handle one real exclusion honestly

Find (or, if you were lucky and had none, deliberately induce by killing a job) one mechanical failure. Exclude it with documentation: the reason, the evidence, and the rerun’s run_id. Then find one disappointing-but-valid result and keep it, with a note.

Reflect: State both decisions in one table (excluded-with-reason vs kept-despite-ugly). Why is the second row the one that proves your integrity, not the first?

Rep 7 — Run the AI-instrument calibration rep (if an LLM is in your method)

For one experimental call, log the exact model version string, provider, date/time, verbatim prompt (or its hash), and temperature. Then verify a sample of the model’s outputs against ground truth before letting any of them into your dataset.

model_version: claude-opus-4-8   provider: anthropic   date: 2026-11-22T09:30Z
temperature: 0.0   prompt_hash: p_4471
verified: 30/30 outputs hand-checked vs human labels before aggregating

Reflect: Long-context fabrication rises with input length (arXiv:2603.08274). In your sample, did the model invent anything? What’s your rule for keeping a model output as data? (If no LLM instrument: instead, write the determinism guarantees of your pipeline — seeds set where, and whether your results reproduce bit-for-bit on a rerun.)

Rep 8 — Audit for dirty-tree runs

Scan your final log for any row with dirty=true.

awk -F, 'NR==1 || $4=="true"' results/run_log.csv

Reflect: How many dirty rows? Each one is unreproducible — you no longer know exactly what code ran. What’s your plan: rerun them clean, or document them as excluded? (There’s no third honest option.)

Rep 9 — Check seed continuity

Confirm your seed sequence has no silent gaps. Every missing seed must trace to a documented exclusion + rerun, not a quiet deletion.

Reflect: Sort your runs by condition and seed. Is seed 3 there? Is every seed there for every condition? Where the Data Integrity Ledger widget would ask “where is seed N?”, can you answer it in writing for your own data?

Rep 10 — Pin and archive the environment

Freeze your environment and tag a release.

conda env export > environment.yml     # or: pip freeze > requirements.txt
git tag -a results-v1 -m "raw results dataset, full run complete"

Reflect: If you deposited this tag to Zenodo today, would the minted DOI point at a setup a stranger could actually rerun? Name the one thing still missing (a README rerun command, a data download script, a CUDA version note).

Done? One Last Thing.

Run code/repro_checklist.txt against your real dataset, top to bottom, and fix every unchecked box. Then write the one-paragraph honesty statement that will open your deliverable: in plain language, what you ran, how many runs landed, what you excluded and why, what you kept despite disliking it, and whether a stranger could reproduce any number you report. If you can write that paragraph truthfully and it still makes you a little uncomfortable — good. That discomfort is the eighth commandment doing its work. That paragraph is the spine of Project 11.

Up next: Project 11