Chapter 10 — Pilot Experiments

“Accounting for the variance induced by these sources [data sampling, initialization, hyperparameters] is necessary to draw reliable conclusions from a benchmark.” — Bouthillier et al., Accounting for Variance in Machine Learning Benchmarks (MLSys 2021)

“For whoever has despised the day of small things shall rejoice…” — Zechariah 4:10 (ESV)

Why This Matters

Last week you built the environment — pinned, seeded, version-controlled, a README a stranger could follow (Chapter 9). The machine is ready. Every instinct now says: launch the full experiment. You have the proposal (Chapter 8), the design (Chapter 7), the compute. Go.

Don’t. Not yet.

This week you run a pilot — a deliberately tiny version of your real experiment. Ten percent of the data. Three seeds, not thirty. One configuration, not the grid. The pilot is not a result and it is not a rehearsal you skip when you’re confident. It is the single cheapest place in the entire research process to discover that your pipeline is broken. A pilot that costs you an afternoon can save you a thousand dollars of GPU time, three weeks of analysis, and — far worse — a finished paper built on a number that was never real. The reproducibility crisis is not mostly fraud. Baker’s 2016 Nature survey of 1,576 researchers found that more than 70% had failed to reproduce someone else’s experiment and more than half had failed to reproduce their own. Most of that is not lying. It is pipelines that quietly did the wrong thing, and nobody piloted hard enough to catch it.

Here is the AI thread, from both sides. As a tool, an LLM or coding agent will happily write your entire training loop, your metric, and your data loader in one go — and any of those can contain a leak, a metric pointed the wrong way, or an off-by-one in the split that the model states with total confidence. The pilot is where you, the human, verify what the agent generated by running it small and watching the numbers. As a workload you study, if your domain project is an AI system — an LLM agent for code review, a text-to-SQL optimizer, an RL congestion controller — the pilot is where you discover that your benchmark is contaminated, your prompt template silently truncates, or your “improvement” is within seed noise. Either way the rule from the spine holds: AI accelerates; the human decides, verifies, and is accountable. A pilot is that verification made cheap.

And the apologetic question Zechariah hands us: Why not despise the day of small things? There is real pride in wanting to skip the small run and go straight to the impressive one. The discipline of the pilot is a discipline of humility — of doing the small, unglamorous, un-publishable work that makes the large work trustworthy. We’ll develop that in §10.7. For now: the small numbers are not beneath you. They are the foundation everything else stands on.

Coach’s Note — A pilot is to your experiment what a layup line is to a game. Nobody buys a ticket to watch the warm-up. But the player who skips it is the player who pulls a hamstring in the first quarter. You do the small reps because the big game matters.

10.1 — What a Pilot Is (and Is Not)

A pilot experiment runs your exact pipeline end to end, at a fraction of the scale, for one purpose: to validate the pipeline, not to measure the effect.

The distinction is everything. If you treat the pilot’s number as a result, you will be tempted to keep it when it looks good and to “fix” the pipeline when it looks bad — which is exactly backwards. The pilot’s job is to break, cheaply, so the full run doesn’t break expensively. A pilot that surfaces three red flags was a success.

The Fact Brief frames the value precisely: the pilot validates the pipeline — data loading, splits, metrics, seeds, compute budget — before full execution, and it directly catches the two failure modes you studied in Chapter 7: leakage (Kapoor & Narayanan 2023) and run-to-run variance (Bouthillier et al. 2021).

10.2 — A Debugging Methodology for Experiments

Debugging an experiment is not like debugging an app. An app crashes; you get a stack trace. An experiment succeeds and hands you a number — a wrong number, with no error at all. The bug is silent. So you need a methodology built for silent bugs.

Work from the trivial outward. The order matters:

1. Can it overfit a tiny batch? Take 8–16 examples and train until the model memorizes them (training loss → ~0). If it can’t overfit 16 examples, your model/loss/optimizer wiring is broken. This is the fastest possible “is the gradient even flowing” check.
2. Does a trivial baseline behave? Run “always predict the majority class” or “random.” If your real method can’t beat it on the pilot, stop. (See code/pilot_sanity_check.py.)
3. Is the metric in range and pointed the right way? Accuracy in [0,1]; perplexity ≥ 1; lower-is-better vs higher-is-better — confuse the direction and you’ll “improve” by getting worse.
4. Is the loss actually moving? Flat loss = nothing is learning. Loss → NaN = exploding gradients, a bad learning rate, or a log of zero.
5. Only now, the real comparison. Method vs strong baseline, on the pilot scale, across a few seeds.

Coach’s Note — When a result surprises you, your first hypothesis should be “my pipeline is wrong,” not “I made a discovery.” Ninety-nine times out of a hundred, the surprising number is a bug. The hundredth time is a paper. Earn the hundredth by ruling out the ninety-nine.

If your project uses an LLM as an instrument (e.g., LLM-as-judge, or an agent that generates code), add a sixth check: does the model output even parse? A pilot constantly reveals that 8% of responses didn’t return valid JSON, or the prompt truncated at the context limit, and your parser silently scored those as failures. Catch that on 50 examples, not 50,000.

10.3 — Sanity-Checking the Metric and the Baseline

Two numbers can lie to you on a pilot: the metric and the baseline. Pin both down.

The metric. Compute it three ways and make sure they agree: by hand on 5 examples, with your code, and (where one exists) with a library implementation (sklearn.metrics, evaluate, torchmetrics). If your hand calculation and your code disagree, your code is wrong. Confirm the range (the Pilot Sanity Checker flags out-of-range values) and the direction. A shocking number of “results” are a minimization metric reported as if higher were better.

The baseline. A method that beats nothing is not a method. You need two baselines:

Coach’s Note — The most dangerous result in research is “my method beats a baseline I implemented badly.” Reviewers know this trick. Your strong baseline must be a baseline you’d be embarrassed to lose to — run it as carefully as your own method, or more so.

For applied-AI projects the trivial baseline is often humbling and clarifying. Before you claim your fine-tuned model helps, check: does a well-written prompt on the base model already solve it? The Fact Brief’s decision ladder — Prompt → RAG → Fine-tune → Distill — is also a baseline ladder. Pilot the cheap rung first.

10.4 — Catching Leakage and a Broken Split

Data leakage is the leading silent killer of CS results. Kapoor & Narayanan’s 2023 survey in Patterns documented leakage across 294 papers in 17 fields. The classic bug is mechanical and easy to miss: you scale features or select features using the whole dataset, then split — so the test set has effectively seen itself.

Run code/leakage_demo.py and watch it happen on pure-noise labels:

# WRONG — scaler fit on all of X, then split (test rows influenced the fit)
scaler = StandardScaler().fit(X)
Xtr, Xte, ytr, yte = train_test_split(scaler.transform(X), y, test_size=0.25)

# RIGHT — split first, fit the scaler on TRAIN ONLY
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25)
scaler = StandardScaler().fit(Xtr)
Xte = scaler.transform(Xte)

On random labels the clean version sits near 0.50, as it must. If your real pipeline shows a number that feels too good, run this pattern of suspicion against it.

Two more leakage forms to pilot for:

• Duplicate / near-duplicate samples that land in both train and test (common in scraped datasets). The Sanity Checker’s split-overlap check catches exact ID overlap; near-duplicates need a hash or embedding pass.
• Benchmark contamination (LLMs): the test items are in the model’s pretraining data. You cannot fully fix this, but you can mitigate by preferring contamination-resistant benchmarks — as of 2026, LiveCodeBench, LiveBench, MMLU-Pro, FrontierMath are the usual citations — and by reporting the risk honestly. (The exact contamination percentages floating around the literature are contested and mis-paired across papers; cite a specific number only to its specific source.)

Coach’s Note — Leakage is the bug that rewards you for having it. It makes your number better, so nothing in your workflow complains. The only defense is a deliberate, suspicious pilot — and a sklearn Pipeline object, which fits every transform inside cross-validation folds so the test fold never touches the fit.

10.5 — Measuring on a Small Run: Variance and Compute

One run is an anecdote. Bouthillier et al. (2021) showed that deep-learning benchmark results vary substantially with data sampling, initialization, and hyperparameters — enough that a single-run “we beat the baseline” is statistically empty. The pilot is where you confront this for the first time, cheaply, by running a handful of seeds and reporting the spread, not a point.

import numpy as np
scores = np.array([0.81, 0.79, 0.83, 0.80, 0.82])   # one number per seed
print(f"mean={scores.mean():.3f}  spread={scores.ptp():.3f}  n={len(scores)}")
# If your "improvement" over the baseline is smaller than this spread,
# you have not yet shown an improvement.

You won’t do formal significance testing here — that’s Chapter 12 — but the pilot answers the prior question: is my effect even bigger than the noise? If three pilot seeds already overlap the baseline, scaling to thirty seeds will not save you; rethink the method.

The compute extrapolation. Record what the pilot cost — GPU-hours, wall-clock, or API calls and dollars — and extrapolate honestly to the full run. A 10% pilot does not always imply 10× cost: attention is quadratic in sequence length, some algorithms are super-linear in data, and API costs scale with tokens, not just calls. Write the estimate down (the report template, code/pilot-results-report-template.txt, has a row for it) and check it against your budget before you commit. A no-go because of cost is a perfectly good pilot outcome.

10.6 — The NeurIPS-Style Pilot Checklist

The pilot is the right time to fill out a reproducibility checklist, because every “No” is still cheap to fix. The NeurIPS Paper Checklist descends from Joelle Pineau’s ML Reproducibility Checklist (piloted 2018, mandatory at NeurIPS 2019). At NeurIPS a missing checklist is a desk reject, and four of its items are exactly what a pilot exists to answer:

• Error bars / variability across seeds — you have them now (§10.5).
• Data splits and hyperparameters, and how they were chosen — your pilot config records them.
• Reproducibility — your Chapter 9 repo + commit hash.
• Compute — your extrapolation from §10.5.

Use code/neurips-style-pilot-checklist.txt and answer for the pilot. The stop rule is the point: if those four items are “No” on the pilot, do not start the full run.

Coach’s Note — The NeurIPS checklist item numbering and wording change every cycle. Treat the file in this chapter as the spirit of the thing, and read the actual checklist for your target year before you submit. The discipline is timeless; the line numbers are not.

This checklist also belongs in your research portfolio (the running requirement since Week 1), filed next to your experimental notebooks and your draft manuscript. A graded pilot checklist now is a paragraph of your Methods section later.

10.7 — The Day of Small Things (Apologetic)

Why not despise the day of small things? Zechariah speaks the line to a discouraged people rebuilding the temple after exile. The foundation they had laid looked pitiful next to the memory of Solomon’s; the older men wept when they saw it (Ezra 3:12). And God’s word through Zechariah is not the small thing doesn’t matter — it is do not despise it, because the small foundation is the thing the whole house will rest on. “The hands of Zerubbabel have laid the foundation of this house; his hands shall also complete it” (Zech. 4:9, ESV).

There is a temptation native to the researcher, and it is a form of pride: the small run is beneath me. I know my method works. Let’s see the impressive number. The pilot feels like the warm-up nobody applauds. But the discipline of running it is the discipline of humility — a willingness to be shown wrong by sixteen examples and a noise baseline before the world ever sees the work. The reproducibility crisis is, at root, a crisis of skipped small things: pipelines nobody piloted, baselines nobody ran carefully, error bars nobody computed because one run already looked good.

This connects to the spine of the whole course. Research is “the glory of kings to search things out” (Prov. 25:2, ESV) — and searching out the truth means being willing to find that your result was a leak. Honesty with a pilot is the same virtue as honesty with the full dataset (Ex. 20:16, the verse over Chapter 11): you bear true witness to what actually ran, even when the small numbers embarrass you. The researcher who does the unglamorous small work, who lets the pilot break and is glad it broke cheaply, is practicing exactly the faithfulness in little things that the work of the day of small things asks for. Do not despise it. The house rests on it.

10.8 — Interactive Lab: Pilot Sanity Checker

Below this chapter on the website you’ll find the Pilot Sanity Checker — a panel that runs a simulated pilot and asks you to spot the red flags before “committing compute.”

Use it like this:

1. Generate a simulated pilot. The widget hands you a metric value, a baseline, the train/test split, and a few per-seed scores.
2. Before you click “scale it,” flag what’s wrong: a metric out of range, a method that only beats the baseline by luck, leakage between splits, a seed spread that swallows the effect.
3. The widget scores your judgment and explains each flag — the same four-plus checks in code/pilot_sanity_check.py.

What it teaches: the reflex of suspicion. After a dozen rounds you’ll see “accuracy = 0.99 on the pilot” and think leakage, not victory — which is exactly the instinct §10.2 is trying to build. The widget is a flight simulator for the most expensive mistakes in your project, at zero cost.

10.9 — Common Pitfalls

Pitfall: Treating the pilot number as a result. Example: Your 10% pilot shows 84% accuracy, you write it into a slide as “our method achieves 84%.” Fix: The pilot measures the pipeline, not the effect. Label every pilot number “pilot — not a result” in your notebook. The real number comes from the full run in Chapter 11.

Pitfall: The too-good number you don’t investigate. Example: Accuracy jumps to 0.99 and you celebrate instead of suspecting leakage. Fix: Make “too good” a trigger for more scrutiny, not less. Run code/leakage_demo.py’s pattern of thinking; check for duplicate samples across the split; verify the metric on 5 hand examples.

Pitfall: One seed, one config, one conclusion. Example: You run once, beat the baseline by 0.6 points, and decide it works. Fix: Run a handful of seeds even on the pilot. If your margin is smaller than the seed spread (§10.5), you have nothing yet. Report the spread, not the single best run.

Pitfall: A baseline you implemented to lose. Example: Your “strong baseline” uses default hyperparameters while your method got tuned. Fix: Tune the baseline as hard as your method, or harder. The reviewer’s first question is “did you give the baseline a fair shot?” Answer it before they ask.

Pitfall: Trusting an agent-generated pipeline because it ran without errors. Example: An LLM wrote your data loader and metric; it executes cleanly, so you assume it’s correct. Fix: “Ran without crashing” ≠ “computed the right thing.” Silent bugs don’t crash. Pilot it: overfit 16 examples, check the metric by hand, verify the split. The human verifies what the agent generated — that’s the non-negotiable loop.

Pitfall: Skipping the compute extrapolation. Example: You scale straight to the full grid and burn the budget at 60% completion. Fix: Measure the pilot’s cost, extrapolate (watching for super-linear scaling), and confirm against budget before launching. A no-go on cost is a successful pilot.

Pitfall: No provenance on the pilot run. Example: The pilot’s numbers exist but you can’t say which commit or config produced them. Fix: Record commit hash, config file, and seeds with every pilot number, exactly as you will for the full run (Chapter 11). A number without provenance is not yet evidence.

10.10 — Reps

The work this week lives in the exercises — 8–12 graduate reps that run a real pilot on your project and produce the evidence for this week’s deliverable. A preview:

• Overfit sixteen examples of your own data and confirm your model can memorize them.
• Run the trivial baseline on your pilot and write down the number you must beat.
• Plant a leakage bug in your own pipeline, watch the metric inflate, then fix it.
• Pilot across 3–5 seeds and report the spread next to your effect.
• Extrapolate the compute from pilot to full run and make a go/no-go call.

Then attempt the Done? One Last Thing. capstone, which assembles the Pilot Results Report. Don’t forget the on-page Check Your Reps quiz below the chapter — five quick questions to confirm the ideas stuck before you spend compute.

10.11 — This Week’s Deliverable

This week produces the Pilot Results Report — see Project 10. It is the artifact that says, on the record: here is what ran small, here is what the small numbers mean, and here is exactly what I will fix before I spend the full budget. It is not graded on whether your method “won” the pilot (it shouldn’t even try to win yet) — it’s graded on the rigor of your sanity-checking and the honesty of your go/no-go call.

Bank the report, the checklist, and the pilot config in your research portfolio and Git repository. If your project investigates how modern AI advances your chosen domain, this is where you first find out whether the AI method actually clears its baseline — or whether the gain you expected was seed noise all along. Better to learn it now, on the pilot, than in the discussion section of a paper that’s already wrong.

For setup details (Colab/Codespaces or a local venv), see Appendix A; for the literature and matrix tools, Appendix B; for the AI integrity and disclosure rules you must follow in the report, Appendix C; for terms, Appendix D.

10.12 — Coach’s Final Word

The pilot is the most professional thing you’ll do all semester, and the least glamorous. No one will read your pilot report after the paper is done. But it is the hour where you decide whether the next month of work rests on sand or on rock. The amateur runs the big experiment and hopes. The researcher runs the small one and checks — overfits sixteen examples, beats a noise baseline, plants a leak and fixes it, counts the seeds, costs the compute, and only then commits.

Do not despise the day of small things. Run it small. Let it break cheaply. Bear true witness to what you find. Then — and only then — scale.

See you on Monday.

Up next: the exercises for the reps · Project 10 for the Pilot Results Report · then Chapter 11, where you run the real thing and keep the integrity of every number.