Chapter 9 — Reps

Every rep this week builds a piece of your own reproducible research environment. By Friday you will have a repository a stranger could clone and run — which is exactly this week’s deliverable. No toy projects; the gym is your actual thesis foundation.

Ground rules.

• Work in your research repository, on the domain and question you locked in Weeks 1–8. If you do not yet have a repo, Rep 1 creates it.
• Pin to exact versions. “It probably still works” is not a rep — pip freeze is.
• Use AI freely to scaffold boilerplate, then run every generated command on a clean environment before you commit it. You certify what the model writes.
• For each rep, write 2–4 sentences of reflection in your portfolio’s docs/lab-notes.txt. The reflection is graded as much as the artifact.
• Commit after every rep with a message that says what changed. Your Git log is your lab notebook this week.

Rep 1 — Scaffold the repository

Create the standard layout and make the first commit. Use the structure from §9.1.

mkdir -p my-research-project/{data/{raw,processed},src,notebooks,results,docs}
cd my-research-project
git init
printf "# %s\n\nReproducible research environment.\n" "My Research Project" > README.txt
touch src/config.py data/README.txt docs/lab-notes.txt
git add -A && git commit -m "scaffold: standard research project layout"

Reflect: Which directory will hold your source of record (the number that goes in the paper) versus your scratch exploration? Write the one-sentence rule you will hold yourself to.

Rep 2 — Write a research .gitignore

Adapt the research .gitignore from §9.2 to your stack. The job: keep the recipe, exclude the output, never commit secrets or large data.

# Start from a stack-appropriate template, then prune to YOUR project
curl -sL https://www.toptal.com/developers/gitignore/api/python > .gitignore
printf "\n# research artifacts\ndata/raw/\ndata/processed/\nresults/checkpoints/\n*.ckpt\n.env\n" >> .gitignore
git add .gitignore && git commit -m "chore: research .gitignore"

Reflect: Name one file in your project that is tempting to commit but shouldn’t be (a dataset, a checkpoint, a .env), and how a collaborator would get it instead.

Rep 3 — Pin your environment at the right layer

Choose the cheapest layer that lets a stranger rerun your project (§9.3) and pin it. Then prove it by reconstructing in a clean environment.

# Layer 1 (pure Python):
python -m venv .venv && source .venv/bin/activate
pip install <your exact deps>
pip freeze > requirements.txt
# Prove it reconstructs:
deactivate && python -m venv /tmp/clean && source /tmp/clean/bin/activate
pip install -r requirements.txt && python -c "import <your_pkg>; print('ok')"

If you have native deps (CUDA/compilers), do this with conda env export --no-builds > environment.yml and conda-lock instead.

Reflect: Which layer did you pick and why — what specifically about your project’s result depends on it? If you chose pip freeze, justify that Docker would be ceremony, not rigor.

Rep 4 — Centralize configuration

Put every hyperparameter, path, split ratio, and the seed in one config.py or config.yaml. No magic numbers scattered across files.

# src/config.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Config:
    seed: int = 1337
    test_size: float = 0.2
    data_path: str = "data/processed/dataset.csv"
    # ... every knob your experiment turns
CONFIG = Config()

Reflect: A reviewer should be able to read your entire experimental setup in this one file. Read yours back — is anything material still hiding as a literal in your code?

Rep 5 — Write set_all_seeds() and prove repeatability

Adapt code/set_seeds.py to cover every randomness source your stack uses. Then run the same command twice and show the outputs are identical.

export CUBLAS_WORKSPACE_CONFIG=:4096:8   # if you use CUDA
python src/run_experiment.py --seed 1337 > results/run_a.txt
python src/run_experiment.py --seed 1337 > results/run_b.txt
diff results/run_a.txt results/run_b.txt && echo "REPEATABLE"

Reflect: Repeatability (same seed → same answer) is not the same as a result. In one sentence, why is “I set seed 42 and beat the baseline” statistically empty (cite Bouthillier et al. 2021)?

Rep 6 — Multi-seed readiness

You won’t run the full multi-seed study until Chapter 12, but make it a one-line change now. Confirm your runner accepts a seed and that looping over seeds is trivial.

for s in 1 2 3 4 5; do
  python src/run_experiment.py --seed "$s" --out "results/seed_$s.json"
done

Reflect: When you vary the seed across these runs, what do you expect the spread to look like for your metric — tiny, or alarming? What would a huge spread tell you about your claim?

Rep 7 — Document one dataset with provenance

Fill out data/README.txt for one dataset using the four provenance questions from §9.5, and start a Datasheet from code/DATASHEET_template.txt. Record a checksum.

sha256sum data/raw/dataset.csv   # macOS: shasum -a 256
# paste the hash into data/README.txt so a stranger can verify the same file

Reflect: Where did your data actually come from — URL, DOI, access date? If your answer is “a colleague sent it,” what is your plan to make that a real, citable provenance?

Rep 8 — Hunt and remove leakage

Audit your preprocessing for the leakage bug from §9.5: any transform fit on data that includes the test set. Refactor so transforms fit on train only — ideally inside a Pipeline.

from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
                                          random_state=CONFIG.seed, stratify=y)
pipe = make_pipeline(StandardScaler(), clf).fit(X_tr, y_tr)  # no leak

Reflect: Did you find a leak? If your study uses an LLM on a static benchmark, also note: could the model have seen your test set in pretraining (contamination), and how would you check?

Rep 9 — Write the reproducible README

Using code/README_template.txt, write the README that takes a stranger from git clone to your headline number: what it produces, environment, data download + checksum, the exact reproduce command, expected output, license.

Reflect: List the two things you had to add that you “just knew” and had never written down. Those are the lines that save your reviewer.

Rep 10 — Score yourself on the widget

Open the Reproducibility Checklist widget on the chapter page. Toggle the six practices honestly for your repository — code public, environment pinned, seeds set, data versioned/documented, exact commands in a README, results archived with a DOI. Record the score and the “fix first” item it names.

Reflect: What did the widget say to fix first, and what is the one commit this week that closes it? Make that commit before the deliverable is due.

Rep 11 — (Stretch) Containerize or archive

Pick one, whichever your project actually needs:

• Docker — write the Dockerfile (code/Dockerfile), docker build, and run your experiment inside the container.
• Archival — connect the repo to Zenodo, push a tagged release, and confirm you get a DOI (and the auto-linked Software Heritage SWHID).

Reflect: Which did you choose, and what reproducibility risk does it retire that the previous reps did not?

Done? One Last Thing.

The clean-machine test — the capstone rep. This rehearses the deliverable’s hardest grading criterion directly: could a stranger rerun this?

On a machine (or a fresh container, or a Codespace) that has never seen your project, do only what your README says — no shortcuts, no memory, no editing files. Clone, build the environment, download the data, run the reproduce command. Time it. Write down every place you had to deviate from the README or reach for knowledge that wasn’t written down.

# In a fresh Codespace or container, README-only:
git clone <your-repo-url> && cd my-research-project
# ...follow the README exactly...

Every deviation is a missing line. Fix the README until a clean machine goes start-to-finish untouched. That repository — the one that survives a stranger — is what you submit.

Up next: Project 9 — the Reproducible Experiment Setup.