Chapter 10 — Reps

This week you run a real pilot on your own research project — small, suspicious, and honest — and produce the evidence for the Pilot Results Report. Every rep moves your project forward; none is busywork.

Ground rules

• Use your own project. Not a toy from a tutorial — the experiment you designed in Chapter 7 and set up in Chapter 9. If your full experiment isn’t runnable yet, that is this week’s discovery.
• Pilot scale, always. Roughly 10% of the data (or 50 LLM calls, or 3 seeds). If a rep takes more than an hour, you’ve scaled it too big.
• No pilot number is a result. Label every number in your notebook “pilot — not a result.” Results come in Chapter 11.
• Record provenance with every run: commit hash, config file, seed. A number without provenance is not evidence.
• AI policy: use a coding agent freely to write pipeline code, but you verify every line by running it small. Disclose substantive AI assistance per Appendix C. “The model wrote it” is never a defense.

Rep 1 — Carve out the pilot

Create a pilot config that runs your exact pipeline on ~10% of the data, a few seeds, one configuration. Commit it.

# example: a 10% subset + a short run, recorded by commit
git checkout -b pilot
python train.py --config configs/pilot.yaml --subset 0.10 --seed 0
git add configs/pilot.yaml && git commit -m "Add pilot config (10% data)"
git rev-parse --short HEAD   # record this hash in your report

Reflect (3–4 sentences): What did you have to change to shrink the run? Did anything break when you shrank it (a hard-coded batch size, a path)? That breakage is a finding.

Rep 2 — Overfit sixteen examples

Take 8–16 training examples and train until your model memorizes them (training loss → ~0).

python train.py --config configs/pilot.yaml --subset-n 16 --epochs 200 --eval-on-train
# Expect training accuracy ≈ 1.0 / training loss ≈ 0. If it can't, your wiring is broken.

Reflect: Could it overfit 16 examples? If not, the bug is in your model/loss/optimizer wiring — before any data question. Write down what you found and fixed.

Rep 3 — Run the trivial baseline

Compute the number your method must beat for free: majority-class, random, or “predict the mean.” Borrow the logic from code/pilot_sanity_check.py.

Reflect: What is your trivial baseline’s score on the pilot? Does your method beat it? If it doesn’t, stop everything else and debug — that’s the whole point of the rep.

Rep 4 — Reproduce a strong baseline

Run the best prior method you can actually execute (the one you cited in your lit review) on the pilot, tuned as carefully as your own method.

Reflect: What is the strong baseline’s pilot score, and how hard did you try to make it win? Name one way you could be giving your own method an unfair advantage, and how you neutralized it.

Rep 5 — Verify the metric three ways

Compute your primary metric (a) by hand on 5 examples, (b) with your code, (c) with a library (sklearn.metrics, evaluate, or torchmetrics).

from sklearn.metrics import f1_score
print("library:", f1_score(y_true[:5], y_pred[:5], average="macro"))
# compare to your hand calculation and your own implementation

Reflect: Did all three agree? Confirm the metric’s range and its direction (higher- or lower-is-better). Catch a direction error here and you’ve saved your paper.

Rep 6 — Plant a leak, watch it, fix it

Deliberately introduce the classic leakage bug into your own pipeline (scale/feature-select before the split), run it, then fix it. Use code/leakage_demo.py as the pattern.

Reflect: How much did the leaky number inflate over the clean number on your data? Now audit your real pipeline: are there any other places the test set could influence the fit (duplicate samples, contaminated benchmark)?

Rep 7 — Pilot across seeds and report the spread

Run your pilot under 3–5 seeds. Report mean and spread, not a single number.

import numpy as np
scores = np.array([...])   # one per seed
print(f"mean={scores.mean():.3f}  spread={scores.ptp():.3f}  n={len(scores)}")

Reflect: Is your method’s margin over the baseline larger than the seed spread? If not, you do not yet have an effect — and thirty seeds won’t fix a method that overlaps the baseline at three.

Rep 8 — Extrapolate the compute and decide

Measure the pilot’s cost (GPU-hours, wall-clock, or API dollars) and extrapolate to the full run, watching for super-linear scaling. Fill the compute rows in code/pilot-results-report-template.txt.

Reflect: What’s the honest full-run estimate, and is it within budget? If it isn’t, what scales down (fewer seeds? smaller grid? a cheaper rung on the Prompt→RAG→Fine-tune ladder)?

Rep 9 — Fill the NeurIPS-style pilot checklist

Complete code/neurips-style-pilot-checklist.txt for the pilot, honestly.

Reflect: Which items were “No”? Apply the stop rule: are items 4, 6, 7, or 8 failing? If so, you are not cleared for the full run — name what you’ll fix.

Rep 10 — AI instrument sanity (if applicable)

If your project uses an LLM/agent as an instrument (judge, generator, extractor), pilot it on ~50 inputs and measure: what fraction of outputs parsed? truncated? required a retry? Log prompt, temperature, and model version + date (closed models drift — Chapter 11).

Reflect: What was your parse-failure rate, and how were those cases being scored silently? If you don’t use an AI instrument, instead audit one place an LLM coding agent wrote pipeline code and verify it by hand.

Done? One Last Thing.

Assemble the Pilot Results Report (Project 10) from the evidence you just produced: what ran, the small numbers with their seed spread, the sanity checks (use the Pilot Sanity Checker widget to pressure-test your judgment), the red flags you found and what they meant, the compute extrapolation, the fix list, and an explicit go/no-go decision. Commit it to your repo and file it in your portfolio. If you can’t yet write an honest “GO,” that is your most valuable finding this week — name exactly what stands between you and the full run.

Up next: Project 10