Chapter 09 · Week 9

Building the Research Environment

What is built so others can build on it?

Chapter 9 — Building the Research Environment

“An article about computational results is advertising, not scholarship. The actual scholarship is the full software environment, code, and data that produced the result.” — Jonathan Buckheit & David Donoho, paraphrasing Jon Claerbout, WaveLab and Reproducible Research (1995)

“According to the grace of God given to me, like a skilled master builder I laid a foundation, and someone else is building upon it. Let each one take care how he builds upon it.” — 1 Corinthians 3:10 (ESV)


Why This Matters

You crossed the halfway line last week. The proposal is written, defended, revised; your committee — your classmates, your conscience — has signed off on a question worth asking and a plan that could answer it. Phase 1 is over. You spent eight weeks learning to find and frame. Now you do and defend. And the very first act of doing is not running an experiment. It is building the place an experiment can run.

Here is the uncomfortable fact this chapter exists to face. In Baker’s 2016 Nature survey of 1,576 researchers, more than 70% reported having tried and failed to reproduce another scientist’s experiment — and more than half had failed to reproduce their own. Read that twice. The most common victim of an irreproducible result is the person who produced it, six months later, staring at a folder called final_v3_REAL and unable to say which seed, which data split, which version of which library made the number in the paper. That is not a story about fraud. It is a story about engineering, and you have the engineering training to refuse to be in it.

So we build the foundation first. Before one row of results exists, you will stand up a version-controlled repository, pin your environment so it can be reconstructed, set your random seeds so a run is repeatable, document your data so its provenance is known, and write a README a stranger could follow to reproduce your work. That stranger, in six months, is you. This is the week the practicum stops being a set of documents and becomes a running system — the thing a paper points at and says here, run it yourself.

The AI thread runs through all of it, from both sides. As a tool, AI will scaffold your requirements.txt, draft your .gitignore, and explain a conda error faster than Stack Overflow — and it will also confidently invent a flag that doesn’t exist and a package that was never published. As a workload, modern AI is the least reproducible software most of you have ever run: nondeterministic GPU kernels, closed models that silently change under the same name, and benchmark scores that move when nobody touched the code. If your domain investigation touches an LLM, your reproducibility burden is higher, not lower. The human stays in the loop where the judgment lives. AI accelerates the setup; you are the one who can certify that the setup is honest.

And that word — foundation — is this week’s question. Paul writes to a church tempted to follow whichever teacher was loudest that the work is not a performance for an audience but a foundation laid for someone else to build on: “Let each one take care how he builds upon it” (1 Cor 3:10, ESV). A research environment built only to produce your number, today, on your laptop, is advertising. A research environment built so that another person — a reviewer, a successor, a future you — can stand on it and build, is scholarship. What is built so others can build on it? That is not a metaphor we are importing into a CS class. It is the literal definition of reproducible research, and we are going to build it with our hands.

Coach’s Note — Reproducibility is not a virtue you add at the end, like a citation you forgot. It is a property of how you worked from the first commit. You cannot bolt it on in Week 14 when the paper is due — the seed you didn’t set in Week 9 is gone. Treat this week as load-bearing. Everything in Phase 2 stands on it.


9.1 — The Portfolio Becomes a System

You have been carrying a research portfolio since Week 1: a comparison matrix, a Zotero library, a paper draft in Overleaf, your reading notes. Until now it has been a pile of documents. This week it acquires a spine — a Git repository — and the pile becomes a system with a shape another person can navigate.

Here is the layout we will build. It is not the only correct one, but it is a defensible default, and a reviewer who opens it knows immediately where everything lives.

my-research-project/
├── README.txt              # how to reproduce, top to bottom
├── LICENSE                # how others may use it (MIT, Apache-2.0, ...)
├── .gitignore             # what never gets committed
├── environment.yml        # conda environment (pinned)
├── requirements.txt       # pip dependencies (pinned)
├── data/
│   ├── README.txt          # provenance: where each dataset came from
│   ├── raw/               # immutable; never edited by hand (git-ignored if large)
│   └── processed/         # derived from raw by code in src/
├── src/                   # the code that does the work
│   ├── config.py          # all hyperparameters + seed in ONE place
│   └── run_experiment.py
├── notebooks/             # exploration; NOT the source of record
├── results/               # outputs, logs, figures (provenance-stamped)
└── docs/                  # the paper, design doc, DATASHEET

Three rules turn this from a folder into a foundation:

  • raw/ is immutable. You never edit a raw data file by hand. Every transformation is a script in src/ that reads raw/ and writes processed/. If a stranger deletes processed/, your code can regenerate it. If they delete raw/, the data/README.txt tells them exactly where to get it again.
  • Notebooks explore; scripts decide. A Jupyter notebook is a lab bench — wonderful for thinking, terrible as the source of record (cells run out of order, state leaks, the saved output rarely matches the saved code). The number that goes in your paper comes from a script you can run start-to-finish, not from a notebook you ran in a hopeful order.
  • Configuration lives in one place. Every seed, learning rate, split ratio, and path goes in config.py (or a config.yaml), never scattered as literals across ten files. A reviewer should be able to read your entire experimental setup in one file.

Coach’s Note — The single best predictor of whether you can reproduce your own result in six months is whether your configuration lives in one file or is sprinkled through your code as magic numbers. Centralize it now. It costs an hour this week and saves the project in Week 11.


9.2 — Version Control: Git, Done Like a Researcher

You know Git. What you may not have done is use it as a researcher — where the goal is not just “ship the feature” but “be able to say which exact state of the code produced figure 3.” That goal changes how you commit.

The non-negotiable habit: every result is tied to a commit. When you produce a number, you record the commit hash that produced it. We will automate that in Chapter 11, but the discipline starts here.

# Start the repository
git init
git add README.txt .gitignore environment.yml
git commit -m "scaffold: project layout + pinned environment"

# Tag the state that produced a result — so you can return to it exactly
git tag -a v0.1-pilot -m "pilot experiment setup"

# The commit hash that produced a given run (you will log this with each result)
git rev-parse --short HEAD

A .gitignore written for research has a particular job: keep the recipe in version control, keep the output out. Commit code, configs, and small fixed assets. Do not commit gigabyte datasets, model checkpoints, virtual environments, or secrets.

# environments & caches
.venv/
__pycache__/
*.pyc
.ipynb_checkpoints/

# large data & artifacts — tracked by provenance, not by Git
data/raw/
data/processed/
results/checkpoints/
*.ckpt
*.pt

# secrets — never, ever
.env
*.key

For data and model files too large for Git, the standard tools as of 2026 are Git LFS (large files inside a Git workflow) and DVC (Data Version Control — Git tracks small pointer files; the bytes live in remote storage like S3 or Google Drive, and dvc.lock pins exactly which version of the data a commit used). You do not need DVC for a 5 MB CSV. You will want it the moment your dataset is too big to commit but must be versioned alongside the code that consumes it.

AssetToolWhy
Code, configs, small fixed filesGitDiffable, the source of record
Large/binary files in the repo treeGit LFSKeeps the repo cloneable
Datasets that evolve and must be pinned to a commitDVCdvc.lock ties data version to code version
Final archival snapshot for the paperZenodo (DOI)Citable, permanent — see 9.7

9.3 — Pinning the Environment: Three Layers

“It works on my machine” is the confession of someone who did not pin their environment. Reproducibility requires that another person — and future-you — can reconstruct the exact software stack that produced your result. There are three layers, increasing in fidelity and cost.

Layer 1 — pip freeze. The minimum. Capture exact package versions.

# Inside an activated virtual environment
python -m venv .venv && source .venv/bin/activate
pip install scikit-learn==1.9.0 pandas matplotlib
pip freeze > requirements.txt        # exact versions, transitively
# Reconstruct elsewhere:
pip install -r requirements.txt

Layer 2 — conda + a lock file. When you have non-Python dependencies (CUDA, compilers, ffmpeg) or want a single environment spec, use conda. A plain environment.yml records what you asked for; conda-lock records what you got, resolving every transitive dependency to an exact build so the solve is identical across machines.

conda env export --no-builds > environment.yml   # human-readable request
conda-lock -f environment.yml -p linux-64         # exact, reproducible solve

Layer 3 — Docker. The highest fidelity short of shipping the hardware. A container captures the OS-level libraries too — the system libstdc++, the CUDA runtime, the exact gcc. This is the layer that survives “but my OS is different.”

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
CMD ["python", "src/run_experiment.py"]

See code/Dockerfile for the version we will use.

LayerCapturesEffortUse when
pip freezePython package versionsMinutesPure-Python, single platform
conda + conda-lockPython + native libs, exact solveAn afternoonCUDA/compilers, cross-machine
DockerThe whole OS userlandA day”Different OS,” GPU stacks, archival

Coach’s Note — Match the layer to the project. Don’t Dockerize a scikit-learn study that runs in ten seconds on any laptop — that’s ceremony, not rigor. Do Dockerize a CUDA + PyTorch pipeline whose result depends on the GPU driver. The honest answer to “which layer?” is “the cheapest one that lets a stranger rerun it.” That is the right-tool instinct, applied to environments.


9.4 — Determinism: Setting Seeds So a Run Repeats

A program that gives a different answer every time it runs cannot be reproduced — only re-rolled. Most ML and simulation code is nondeterministic by default: weights initialize randomly, data shuffles randomly, GPU kernels accumulate in nondeterministic order. You control this by seeding every source of randomness and, on GPU, by demanding deterministic algorithms.

import os, random, numpy as np

SEED = 1337

def set_all_seeds(seed: int = SEED) -> None:
    random.seed(seed)
    np.random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    try:
        import torch
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.use_deterministic_algorithms(True)
        torch.backends.cudnn.benchmark = False
    except ImportError:
        pass

Two PyTorch gotchas that bite everyone, as of 2026:

  • With CUDA ≥ 10.2, torch.use_deterministic_algorithms(True) will raise a RuntimeError unless you set the environment variable CUBLAS_WORKSPACE_CONFIG=:4096:8 (or :16:8) before the process starts. Set it in your run script, not inside Python.
  • Determinism is not guaranteed across PyTorch releases, across platforms, or between CPU and GPU. A seed makes your runs on your machine repeatable; it does not make a CUDA result bit-identical on someone else’s card. Document the hardware, not just the seed.

The full reference seeding utility is code/set_seeds.py. Now — the deeper point, and the reason this is a research skill and not just a coding trick:

A single seeded run is repeatable, but a single run is not a result. Bouthillier et al. (2021, MLSys, “Accounting for Variance in Machine Learning Benchmarks”) showed that deep-learning results vary substantially with the random sources you just pinned — data sampling, initialization, hyperparameter draws. “We set seed 42 and beat the baseline” is statistically empty: you may have beaten it on the one seed where you happened to win. The discipline is to fix the seed for repeatability of a single run, and to run multiple seeds for the claim. Report the mean and spread across seeds, not the high-water mark. We design that properly in Chapter 12; here, you build the machinery — a seed you control, and a config that makes “run this across seeds 1–10” a one-line change.

Coach’s Note — There are two opposite sins here. One is no seed: your result can’t be repeated at all. The other is cherry-picking the seed: you tried thirty and reported the lucky one. The first is sloppiness; the second is misconduct dressed as a number. Set the seed for repeatability. Vary the seed for honesty. Both, always.


9.5 — Documenting the Data: Provenance, Datasheets, and Leakage

Code without data is half a foundation. The most common reproducibility failure in applied ML is not a missing library — it is a dataset nobody can find, in a form nobody can reconstruct, split in a way nobody documented. Your data/README.txt answers four questions for every dataset you touch:

  1. Where did it come from? A URL, a DOI, a citation, an access date. “From a colleague” is not provenance.
  2. What version / when? Datasets change. Pin the version or the download date.
  3. What did you do to it? Which script in src/ turns raw/ into processed/, and what does it do.
  4. How is it split? Train / validation / test, and the rule that assigns each row — so the split is reproducible and leak-free.

For datasets you assemble or release, the field-standard document is a Datasheet for Datasets (Gebru et al.): motivation, composition, collection process, preprocessing, recommended uses, and known limitations. A starter is in code/DATASHEET_template.txt. It is the data analogue of the reproducible README — it lets someone else decide whether your data is fit for their purpose.

Now the integrity hazard you build the foundation to prevent: data leakage. Kapoor & Narayanan (2023, Patterns) found leakage across 294 papers in 17 fields — results that looked strong and were wrong because information from the test set bled into training. The classic bug is scaling or feature-selecting before the split, so the test data secretly shaped the model.

# WRONG — the scaler sees the test set; leakage inflates your score
from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X)        # fit on ALL data — leak!
X_train, X_test = split(X_scaled)

# RIGHT — fit only on train; a Pipeline makes leakage hard to commit by accident
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=SEED, stratify=y)
pipe = make_pipeline(StandardScaler(), clf)         # scaler fit inside CV/fit
pipe.fit(X_train, y_train)

If your domain investigation uses an LLM as the system under study, you inherit a sharper version of this: benchmark contamination. A model may have seen the test set during pretraining — your held-out benchmark was on the web. This is why contamination-resistant benchmarks like LiveCodeBench, LiveBench, and MMLU-Pro exist (as of 2026); if you evaluate a model on a static benchmark older than its training cutoff, you must say so, because the number may be memorization, not capability.


9.6 — The README a Stranger Can Run

The README is the front door to the foundation. A reviewer who cannot reproduce your result from the README will assume — fairly — that you could not either. The test is concrete: could a competent stranger, on a clean machine, go from git clone to your headline number, following only the README, with no questions for you?

A reproducible README has a predictable spine:

# Project Title — one-line description

## What this produces
The result this repository reproduces (e.g., "Table 2 and Figure 3").

## Environment
conda env create -f environment.yml   # or: pip install -r requirements.txt
conda activate my-research-project

## Data
Download the dataset from <DOI/URL> into data/raw/ (see data/README.txt).
Expected: data/raw/dataset.csv, SHA-256 a1b2c3...

## Reproduce
export CUBLAS_WORKSPACE_CONFIG=:4096:8
python src/run_experiment.py --config src/config.yaml --seed 1337
# Outputs results/metrics.json and results/figure3.png

## Expected output
metrics.json: {"accuracy": 0.873, "f1": 0.861}   (± across seeds 1–10)

## Cite / License
MIT. If you use this, cite <preprint/DOI>.

A working skeleton is in code/README_template.txt. Notice the SHA-256 line: a checksum on the raw data is how a stranger knows they downloaded the same file you used. Notice the explicit export of the CUDA variable: you are doing the reader’s debugging before they hit the error. That is what “take care how he builds upon it” looks like in a README.

Coach’s Note — The cruelest, most useful test of a README is to run it yourself on a borrowed laptop you’ve never installed your project on. You will be horrified at what you “just knew” and never wrote down. Do this before Chapter 10. It is an hour that saves your reviewer — and your grade.


9.7 — Archival and Badges: Making the Foundation Permanent

When the paper is done, the repository on your laptop is not a citation — it’s a folder that can be deleted. To make your foundation something others can stand on permanently, you archive a fixed snapshot and get a persistent identifier.

Zenodo (run by CERN and OpenAIRE) mints a DOI for each deposit. Connect your GitHub repository, push a tagged release, and Zenodo archives that exact state and issues a citable DOI. Since its October 2024 integration, Zenodo also auto-deposits public source code to Software Heritage, minting a SWHID (Software Heritage Identifier) bidirectionally linked to the DOI — so your code has both a citable handle and a permanent archival home.

The community also recognizes reproducibility formally. ACM’s Artifact Review and Badging (v1.1) awards three independent badges:

BadgeMeans
Artifacts AvailableThe artifacts are publicly, permanently archived (e.g., a Zenodo DOI)
Artifacts EvaluatedAn independent committee ran them — Functional or Reusable
Results ValidatedAn independent team obtained the results — Reproduced (using your artifacts) or Replicated (without them)

You will not chase a badge in this practicum. But you should build as if you might earn one, because the badge criteria are simply the reproducibility discipline, made explicit. Build for “Available” from Week 9, and “Reproduced” stays possible later.

Coach’s Note — Note ACM’s exact wording, because it is easy to get backwards: Reproduced means an independent team got your result using your artifacts; Replicated means they got it without them. ACM deliberately aligned these with the NISO definitions. If you use the words in your paper, use them the way the community does.


9.8 — AI in the Setup: Accelerant and Liability

Let the AI scaffold the boring parts. It is genuinely good at generating a starter .gitignore, explaining a cryptic conda solver error, drafting a Dockerfile, and writing the boilerplate of set_seeds.py. Use it for that — it is faster than you and rarely wrong about syntax.

Then verify, because the failure modes are specific and this is your name on the foundation:

  • Hallucinated packages and flags. An LLM will confidently write pip install for a package that does not exist, or pass a CLI flag that was removed three versions ago. You run the command and read the error; the model never will.
  • Plausible-but-wrong pins. It may suggest torch==2.x paired with a CUDA version that don’t actually ship together. Version compatibility is exactly the kind of fact LLMs get subtly wrong. Check against the real index.
  • Determinism it can’t promise. Ask for “deterministic PyTorch” and you may get the seeding lines but not the CUBLAS_WORKSPACE_CONFIG requirement — the model gives you the common answer, not the complete one.

And disclosure: if AI substantively generated code or text that ends up in your submission, your target venue’s policy applies (we cover the venue-by-venue rules in Appendix C). As of 2026 the cross-venue consensus is firm — an LLM can never be an author, “the model did it” is never a defense, and substantive generated content gets disclosed. Scaffolding a .gitignore does not require disclosure; generating a core piece of your method does. The line is assistive vs. generative, and you draw it honestly.

Coach’s Note — Here is the spine rule for this chapter. AI can build the scaffold; only you can certify the foundation. The model cannot run your code on a clean machine, cannot know whether your data is leak-free, cannot vouch that the number in your README is the number you actually got. The judgment — is this honest, is this reproducible — never leaves your hands.


9.9 — Interactive Lab: Reproducibility Checklist

Open the Reproducibility Checklist widget embedded directly below this chapter on the site.

The widget is the question of this whole chapter made concrete: could a stranger rerun this? You toggle the practices a reproducible study needs — code is public, environment pinned, random seeds set, data versioned and documented, exact commands in a README, and results archived with a DOI — and the widget weights them by leverage (public code you can’t run is theater; pinned environment and exact commands carry the most weight). It computes a score and, more usefully, names the one thing worth fixing first — the highest-leverage missing piece most likely to break someone else’s attempt to reproduce your work. It also surfaces the PyTorch-determinism trap from §9.4 — the CUBLAS_WORKSPACE_CONFIG=:4096:8 everyone forgets.

Score your own repository, honestly, as it stands today. Whatever the widget names as “fix first” is the highest-value commit you can make this week before the deliverable is due. Do not inflate the self-score; the only person you fool is future-you, in six months, holding final_v3_REAL.


9.10 — The Theology of a Foundation

We named the question at the start: what is built so others can build on it? It is worth slowing down here, because this is the chapter where the apologetic and the engineering are not parallel tracks — they are the same track.

Paul is writing to a church fracturing into fan clubs — “I follow Paul,” “I follow Apollos.” He answers with a building metaphor that refuses the whole frame: “like a skilled master builder I laid a foundation, and someone else is building upon it. Let each one take care how he builds upon it” (1 Cor 3:10, ESV). The point is not that Paul is the great one. The point is that the work was never his to keep. He laid a foundation precisely so that someone he would never meet could build higher than he did. The skill he names — “skilled master builder” — is a skill aimed outward, at the builder who comes next.

That is reproducible research, exactly. A result you can produce once, on your machine, that dies when you close the laptop, is the opposite of a foundation — it is a tower built for an audience, advertising in Donoho’s word. A result built so that a reviewer can verify it, a successor can extend it, and a stranger can stand on it and reach higher — that is a foundation laid with care for the one who builds next. The reproducibility crisis is, underneath the statistics, a stewardship failure: a generation of work laid down so carelessly that the next generation cannot build on it. More than half of researchers cannot rebuild their own foundation. We are called to better — not because reviewers demand it (though they do), but because the work was never only ours.

There is a humility in this the world calls inefficiency. Pinning an environment, writing the README, archiving to a DOI — none of it makes your number better. It is pure service to the builder who comes after, much of whom you will never meet. Lutherans have a word for work like that: vocation — labor done not to be seen but to serve the neighbor God puts in front of you. Your neighbor, this week, is the reader who clones your repository at 2 a.m. and needs it to just run. Build the foundation so they can build on it. Take care how you build.

And there is a sharper edge, because the same chapter that calls you to build well warns about building badly: integrity. A foundation that looks solid but is built on a cherry-picked seed, a leaked split, a hallucinated citation, or a number you wished into the README is not a foundation at all — it is a trap for the next builder. “Let each one take care” is also a warning. The honest seed, the documented split, the verified citation, the number you actually got: these are not bureaucratic hoops. They are how you keep faith with the person who trusts your foundation enough to build on it.


9.11 — Common Pitfalls

Pitfall: The unpinned environment. Example: Your requirements.txt reads scikit-learn with no version. Three months later a clean install pulls 1.10, an API changed, and your script crashes — or worse, silently returns different numbers. Fix: Pin exact versions (pip freeze, conda-lock). Pin the layer the result depends on — if the GPU matters, you need Docker, not pip freeze.


Pitfall: The seed that isn’t set everywhere. Example: You seed numpy but not torch, or not PYTHONHASHSEED, or not the DataLoader workers. Your “deterministic” run still drifts. Fix: Use one set_all_seeds() utility that covers random, numpy, torch, CUDA, and PYTHONHASHSEED, and set CUBLAS_WORKSPACE_CONFIG before the process starts. See code/set_seeds.py.


Pitfall: Leakage hidden in preprocessing. Example: You StandardScaler().fit_transform(X) on the whole dataset before splitting. The scaler learned the test set’s statistics; your accuracy is inflated and the paper is wrong. Fix: Split first, fit transforms on train only — ideally inside a Pipeline so the leak is hard to commit. Document the split rule in data/README.txt.


Pitfall: The notebook as source of record. Example: Your headline number lives in cell 14 of a notebook you ran out of order; nobody — including you — can reproduce the exact state that produced it. Fix: Move the result-producing path into a script you can run top-to-bottom. Keep notebooks for exploration, not for the number that goes in the paper.


Pitfall: Committing the data, secrets, or the venv. Example: You git add . and push a 2 GB dataset, your .env with an API key, and a .venv/. The repo is now huge, the key is leaked forever in history, and the clone takes ten minutes. Fix: Write the research .gitignore first. Track large data with DVC/Git LFS, never the bytes in Git. Rotate any key that ever touched a commit.


Pitfall: Trusting an AI-generated setup without running it. Example: The model gave you a tidy requirements.txt with a torch/CUDA pairing that doesn’t ship together, or a pip install for a package that doesn’t exist. You committed it without testing on a clean env. Fix: Run every AI-generated command on a clean environment before you commit it. The model wrote it; you certify it.


Pitfall: The README only you can follow. Example: It says “run the experiment” but omits the data download, the env activation, and the CUBLAS_WORKSPACE_CONFIG export — all things you “just know.” Fix: Test the README on a machine that has never seen your project (or in a fresh container). Anything you have to do from memory is a missing line.


9.12 — Reps

The work is in the exercises. These reps are not busywork on a toy — every one moves your own research project forward this week. A preview of where you’re headed:

  • Scaffold the repository with the standard layout and a research .gitignore, and make the first tagged commit.
  • Pin your environment at the right layer for your project and prove a clean install reconstructs it.
  • Write set_all_seeds(), wire it into your config, and demonstrate a repeated run gives an identical result.
  • Document one dataset with provenance and a Datasheet starter, and remove any leakage in your preprocessing.
  • Write the reproducible README and test it on a clean machine or container.

The capstone rep rehearses the deliverable directly. And don’t skip the on-page Check Your Reps quiz below the chapter — five questions that catch the misconceptions that quietly wreck reproducibility.


9.13 — This Week’s Deliverable

This week you produce the Reproducible Experiment Setup — a working, version-controlled repository for your research project that a stranger could clone and run. Full spec, rubric, and tiers are in Project 9.

It is the hinge of the practicum. Everything in Phase 1 was paper; from here on, you have a running system. The pilot experiments in Chapter 10 run inside this environment; the full execution in Chapter 11 logs its provenance against this repository; the paper in Week 14 points at it and says here, run it yourself. Build it once, build it right. For setup help — the no-install browser path (GitHub + Codespaces + Colab) and the local path — see Appendix A and the toolkit in Appendix B.


9.14 — Coach’s Final Word

You did not run an experiment this week. You did something harder and more important: you built the place an experiment can live and be trusted. That is the unglamorous work that separates a researcher from a person with a clever idea — anyone can get a number once; only a researcher can hand someone else the means to get it again.

More than half of all researchers cannot reproduce their own work. You are now equipped to be in the other half — not by being smarter, but by being disciplined: a pinned environment, a seed you control, a documented split, a README a stranger can run, a snapshot archived for good. None of it flatters you. All of it serves the next builder. That is the point.

Take care how you build. Someone is going to build on it — a reviewer next month, a successor next year, you in six months. Lay the foundation so they can.

See you on Monday.


Up next: the exercises for the reps · Project 9 for the Reproducible Experiment Setup · then Chapter 10 to run your first pilot inside the environment you just built. Setup help: Appendix A · Appendix B · Appendix C · Appendix D.

Interactive Lab — Week 9
Reproducibility Checklist

Imagine a stranger clones your repo six months from now. Could they reproduce your headline number? Toggle what you've actually done — the tool scores you and tells you the one thing worth fixing first.

0 / 100
Nothing checked yet.
Fix first Start anywhere — every box helps.
PyTorch determinism, the part everyone forgets: a seed alone is not enough on a GPU. Bit-for-bit reruns also need torch.use_deterministic_algorithms(True) and the env var CUBLAS_WORKSPACE_CONFIG=:4096:8 (set before launch). Together with torch.manual_seed(s), numpy.random.seed(s), and random.seed(s), that's the minimum for a deterministic CUDA run — at a small speed cost.
Try: Check only "Code is public" and watch the score barely move — public code you can't run is theater. Now add "Environment pinned" and "Exact commands": those carry the most weight, because they're what actually lets a stranger press play.
Check Your Reps

Check Your Reps — Building the Research Environment

Question 1 of 5
You set seed 42, ran your model once, and beat the baseline. Why is that not yet a defensible research claim?
Why: Setting a seed makes one run repeatable, but a single run is statistically empty; you fix the seed for repeatability and vary it across runs to support the claim with mean and spread.
Question 2 of 5
Which preprocessing pattern avoids the classic data-leakage bug documented by Kapoor & Narayanan (2023)?
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=SEED, stratify=y)
pipe = make_pipeline(StandardScaler(), clf).fit(X_tr, y_tr)
Why: Fitting any transform on data that includes the test set leaks test information into training and inflates the score; split first and fit transforms on train only.
Question 3 of 5
Your result depends on the CUDA runtime and the exact system libraries, and you need a stranger on a different OS to rerun it. Which environment-pinning layer fits?
Why: pip freeze captures Python package versions only; when the result depends on OS-level libraries like the CUDA runtime, Docker is the layer that survives 'but my OS is different.'
Question 4 of 5
In ACM's Artifact Review and Badging vocabulary, what does the 'Reproduced' result mean?
Why: ACM (aligned with NISO) defines Reproduced as an independent team obtaining the result using the authors' artifacts, while Replicated means obtaining it without them.
Question 5 of 5
When you ask an AI assistant to scaffold your research setup, what is the human's non-negotiable responsibility?
Why: AI accelerates the setup but the human stays in the loop where the judgment lives: you run and certify every command, because an LLM can never be an author and 'the model did it' is never a defense.
YOU FINISHED. NICE WORK.