Project 9 — Reproducible Experiment Setup

“According to the grace of God given to me, like a skilled master builder I laid a foundation, and someone else is building upon it. Let each one take care how he builds upon it.” — 1 Corinthians 3:10 (ESV)

Chapter: Chapter 9 — Building the Research Environment Due: End of Week 9 (before the Chapter 10 pilot) Submit: A link to your Git repository (GitHub/GitLab) at the tagged release v0.1-setup, plus a one-page SETUP_MEMO.txt in docs/. Allowed tools: Everything — Git, conda/Docker, your editor, AI assistants for scaffolding. AI policy: Open. Use AI to scaffold boilerplate, but you certify every line. Keep an docs/agent-log.txt noting where AI substantively contributed (per Appendix C). The judgment calls — what layer, what’s leakage, what the README must say — are yours. Weight: Ungraded as a percentage of the course (the graded deliverables are P2/P3/P4, the proposal, P11 results, the draft, and the final), but this is a required gate: the Chapter 11 Experimental Results (15%) is collected from this repository, and unreproducible results there will be marked down. Build it right now or pay for it then.

The Setup

It is six months from today. Your paper is under review at a workshop. A reviewer writes: “The results are interesting but I could not reproduce Table 2 — the repository has no pinned environment, the seed is unstated, and the README assumes the dataset is already present. I cannot recommend acceptance.”

That review is avoidable, and it is avoided this week — not in Week 14 when the seed you didn’t set is already gone. Your job is to stand up the foundation now: a repository for your own research project that a competent stranger, on a clean machine, could clone and run from the README alone, arriving at the same number you did.

This is also the moment your scattered portfolio — matrix, Zotero, draft, notes — becomes one navigable system with a Git spine. From here, the practicum is a running thing.

Learning Targets

You will demonstrate that you can:

• Structure a research repository so another person can navigate it, separating immutable raw data, derived data, source-of-record code, and scratch exploration.
• Pin a software environment at the appropriate fidelity layer (pip freeze / conda-lock / Docker) and prove a clean reconstruction.
• Make a computation repeatable by controlling every source of randomness, and articulate why repeatability is necessary but not sufficient for a result.
• Document a dataset’s provenance and split so the data pipeline is reproducible and leak-free.
• Write a README that takes a stranger from git clone to your headline number with no questions.
• Reason about which reproducibility investments your specific project needs — and which would be ceremony.

Normal Tier

The deliverable bar for the week: a version-controlled repository a stranger could plausibly run.

Required features

1. Repository + layout. Initialized Git repo with the standard layout (§9.1): data/{raw,processed}/, src/, notebooks/, results/, docs/. A research .gitignore that excludes data, venvs, checkpoints, and secrets.
2. Pinned environment. A requirements.txt (exact versions) and/or environment.yml, at the layer your project needs, with a documented reconstruction command that you have actually run on a clean environment.
3. Centralized config + seeding. All hyperparameters, paths, and the seed in one config.py/config.yaml. A set_all_seeds() utility covering random, numpy, and (if used) torch/CUDA + PYTHONHASHSEED.
4. Data documentation. A data/README.txt answering the four provenance questions (source, version/date, your transformations, split rule) for at least one dataset, with a checksum.
5. Reproducible README. Front-door README: what it produces, environment setup, data download, the exact reproduce command, expected output, license.
6. A runnable entry point. src/run_experiment.py (even a trivial placeholder pipeline is fine this week) that reads the config, seeds, and writes a provenance-stamped output to results/.
7. docs/SETUP_MEMO.txt. One page: which environment layer you chose and why, your weakest reproducibility link (from the widget), and the one commit that most improved your score.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Stronger rigor. Add any of:

• Clean-machine proof. Run the capstone clean-machine test (fresh container/Codespace, README-only) and include the timed transcript in docs/. Show the README survived a stranger untouched.
• Multi-seed readiness with evidence. Run your placeholder pipeline across seeds 1–10 and commit the per-seed outputs plus a one-line summary of the spread — demonstrating you understand a single run is not a claim (Bouthillier et al. 2021).
• conda-lock or Docker. Go one layer higher than the minimum your project strictly needs and justify the cost/benefit in the memo.
• A real Datasheet. Complete a full Datasheet for Datasets (Gebru et al.) for your primary dataset, not just the four-question starter.

Hard Tier (+up to 25% additional extra credit)

Publication-ready foundation — and a judgment a tool cannot make for you.

• Archive for a DOI. Push a tagged release to Zenodo, obtain a DOI (and confirm the auto-linked Software Heritage SWHID), and add the citation to your README. Your foundation is now permanent and citable.
• The Reproducibility Risk Memo (required for Hard). Write a 1–2 page memo, docs/REPRO_RISK_MEMO.txt, that no AI could write for you because it requires knowing your project: identify the three things most likely to make your specific result irreproducible by someone else (e.g., a closed LLM that drifts under the same name, a dataset behind a login that may vanish, GPU-nondeterministic kernels your metric is sensitive to, a benchmark that may be contaminated). For each, state the risk, your current mitigation, and the residual risk you cannot fully retire. Map your repository against the ACM Artifact Available / Evaluated / Results Validated criteria and state honestly which you could earn today and what the gap is.

The Hard tier is the difference between a repo that runs on your laptop and a foundation someone else can build on. The memo is where you prove you understand the difference.

Submission

• Repository URL with a tag v0.1-setup marking the submitted state (git tag -a v0.1-setup -m "reproducible setup deliverable").
• docs/SETUP_MEMO.txt (Normal), plus docs/agent-log.txt if AI contributed substantively.
• Medium/Hard artifacts in docs/ (clean-machine transcript, multi-seed summary, Zenodo DOI, REPRO_RISK_MEMO.txt).

Hints

• Build the .gitignore before your first git add -A. A secret committed once lives in history forever, even after you delete it — rotate any key that ever touched a commit.
• Don’t over-engineer the environment layer. A scikit-learn study that runs in ten seconds does not need Docker; a CUDA + PyTorch pipeline whose number depends on the driver does. Justify your choice in the memo — that reasoning is graded, not the heaviest tool.
• Test the README on something that has never seen your project. You will be amazed at what you “just knew.”
• Let AI draft the Dockerfile and .gitignore; then run the build and read the errors yourself. The model wrote it; you certify it.
• CUBLAS_WORKSPACE_CONFIG=:4096:8 must be set before the Python process starts if you use CUDA determinism — put it in the run script, not in Python.
• The starter files do most of the typing for you: code/set_seeds.py, code/README_template.txt, code/Dockerfile, and code/DATASHEET_template.txt. Adapt them to your stack — don’t ship them verbatim with the placeholders still in.
• If your run_experiment.py has no real pipeline yet, a trivial placeholder that reads the config, seeds, and writes a metrics.json is fine this week. The grade is on the foundation, not the result — the result comes in Chapters 10–11.
• No-install path: GitHub for the repo, a Codespace as your clean machine, Colab for compute, Overleaf for the paper — all browser-based. See Appendix A.

A note on the AI thread

If your investigation puts a model under study — an LLM agent for code review, a vision model, an RL policy — your reproducibility burden is higher, not lower, and the memo should say so. Closed models drift silently under the same name, GPU kernels are nondeterministic, and a static benchmark may be contaminated (the model saw your test set in pretraining). Record the model version and date the way you record a seed and a commit. A “deterministic” experiment on a moving model is a contradiction you must disclose, not hide.

What Mastery Looks Like

A grader clones your repository on a machine they’ve never used, reads only the README, and ten minutes later is looking at your headline number — no email to you, no guessing, no “oh you also have to…”. They open SETUP_MEMO.txt and find a researcher who chose their environment layer rather than defaulting to it, who knows their own weakest link, and who tells the truth about it. That is a foundation someone else can build on.

Coach’s Note — The temptation this week is to make it look reproducible — a tidy README, a requirements.txt — without it being reproducible. The only test that matters is the clean-machine run. Do it before you submit, not after a reviewer does it for you. The reviewer is less forgiving than I am.

When You’re Done

• Repo initialized, standard layout, research .gitignore in place — no data/secrets/venv committed.
• Environment pinned and reconstruction command verified on a clean env.
• One config file holds every knob + the seed; set_all_seeds() makes a run repeatable.
• One dataset documented with provenance + checksum; split is leak-free.
• README takes a stranger from clone to headline number.
• SETUP_MEMO.txt written; agent-log.txt if AI contributed.
• Tagged v0.1-setup; repo URL submitted.

A theological footnote. “Like a skilled master builder I laid a foundation, and someone else is building upon it. Let each one take care how he builds upon it” (1 Cor 3:10, ESV). The foundation you lay this week is not for you — it is for the reviewer, the successor, the future-you who will build on it. That is why we pin the environment that doesn’t make your number better, document the data that’s “obvious” to you, archive the snapshot that costs an afternoon: not to be seen, but to serve the one who builds next. The reproducibility crisis — more than half of researchers unable to rebuild their own work — is at bottom a stewardship failure. Take care how you build. Build so others can build on it.