Project 7 — Experimental Design Document

“…having followed all things closely for some time past, to write an orderly account for you… that you may have certainty concerning the things you have been taught.” — Luke 1:3–4 (ESV)

Chapter: 7 — Designing a Reproducible Experiment Due: End of Week 7 (before the Week 8 proposal session — this document is the methodology you’ll defend) Submit: A design-doc.txt (or PDF from your Overleaf manuscript’s Methodology section) plus the supporting files in your portfolio repo: configs/experiment-design.yaml, configs/baseline-matrix.csv, a completed reproducibility checklist, and the seeded multiseed_eval.py adapted to your study. Push to your Git repo and submit the commit hash. Allowed tools: Anything. Overleaf, your editor, scikit-learn / SciPy / deep-significance, Zotero for any cited method. LLM assistance permitted as an assistant, not an author. AI policy: Open. You may use an LLM to critique your design, draft prose, or scaffold the harness — but log the model name + version + date, verify every citation it produces against a real index (dblp / Semantic Scholar / publisher), and own every claim. Disclose AI use per Appendix C. The model accelerates; you decide, verify, and are accountable.

The Setup

You are six weeks from running the experiments that will become your results section. The difference between a clean Week 11 and a miserable one is the document you write this week. A good experimental design is the thing you hand a collaborator — or a reviewer, or your future self — that says: here is exactly what I will measure, against what, how many times, and how you would check me.

This is not a class artifact. It is the pre-execution lock of a real study: the same document, in spirit, that a Registered Report submits for in-principle acceptance before any data exists. Write it so airtight that running the experiment in Week 11 is bookkeeping, and so honest that a stranger could rerun it and either confirm or refute you. Next week you defend this design in front of your peers; the proposal is only as strong as the methodology underneath it.

Learning Targets

You will demonstrate that you can:

• Turn a falsifiable hypothesis into a concrete, measurable experimental plan.
• Specify a naive and a strong baseline, and at least one ablation per claim.
• Choose a benchmark and confront contamination / construct-validity threats honestly.
• Define train/validation/test splits with an explicit leakage guard.
• Commit to a primary metric (with direction) and a significance test + effect size before seeing data.
• Build a study that another researcher could reproduce: pinned env, seeds, data version, code commit, LLM instrument pinning, and disclosure.

Normal Tier

A complete, lockable design document another researcher could read and execute.

Required features

1. Claim + hypothesis. Your research question, a falsifiable H1 with a concrete threshold + alpha, and the matching H0.
2. Baselines. A naive/floor baseline and a strong baseline (the current accepted approach — usually what AI is meant to replace), with a sentence on how you’ll tune the strong one fairly.
3. Ablations. At least one ablation per claim, each removing a single component, captured in configs/baseline-matrix.csv.
4. Benchmark + contamination note. The dataset/benchmark named and version-pinned (or a justified custom set), plus a contamination / construct-validity threat if your method is an LLM on public data.
5. Splits + leakage guard. Train/val/test policy, test touched once, “preprocessing fit on train only” stated explicitly.
6. Metric + protocol. Primary metric with direction, secondary metrics, ≥ 5 seeds, mean ± std, and a significance test (ASO / permutation) + effect-size measure chosen in advance.
7. Reproducibility plan. Env pinning, seeds + determinism flags, data version/DOI, code repo + pinned commit, LLM instrument pinning (name/version/date/temp), and the disclosure line.
8. The machine-readable spine. A completed configs/experiment-design.yaml and a completed reproducibility checklist committed to the repo.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Stronger rigor and novelty:

• Pre-register for real. Time-stamp your plan read-only on OSF (or AsPredicted.org) before you collect data, and link it. A genuine, dated lock — not just a Git commit.
• Power / runs justification. Estimate the run-to-run variance (from a tiny pilot or a literature anchor) and justify why 5 seeds — or more — is enough to detect your hypothesized effect.
• Two claims, fully designed. A second, distinct claim with its own baseline comparison and ablation, woven into the same coherent protocol.
• A working, seeded harness. multiseed_eval.py adapted to your real (or faithfully stubbed) pipeline, producing mean ± std and a real test statistic.

Hard Tier (+up to 25% additional extra credit)

Publication-ready judgment a tool cannot supply.

Write a 2–3 page Design Defense Memo that makes the calls only a researcher can make:

• The threat you can’t fully control. Name the single biggest threat to your study’s validity (internal, external, construct, or conclusion) that your design cannot eliminate, and argue honestly for the strongest mitigation you can afford — including what you’ll concede as a limitation.
• The baseline you’d be tempted to weaken, and your guard against it. State where the integrity pressure is highest in your design and the concrete commitment that holds the line (Prov 11:1, ESV).
• The reproduction stranger. Argue, specifically, that a researcher with your repo and one afternoon could reproduce your headline result — and name the one place they’d most likely fail, with your fix. This is the could-I-rerun-this? question a reviewer will ask; answer it before they do.

An LLM can draft prose around your decisions. It cannot decide which threat you accept, which mitigation is worth the compute, or where your own integrity is most under pressure. That judgment — and the accountability for it — is yours.

Submission

• design-doc.txt (or the Methodology PDF from your Overleaf manuscript).
• configs/experiment-design.yaml, configs/baseline-matrix.csv, completed reproducibility checklist, adapted multiseed_eval.py.
• Medium: the OSF/AsPredicted link + power justification. Hard: design-defense-memo.txt.
• Push to your portfolio repo; submit the commit hash (your pre-registration timestamp).
• AI disclosure block per Appendix C.

Hints

• Start with code/experiment-design.yaml. Filling its fields is the assignment; the prose wraps the filled fields.
• The fastest validity check is the on-page Experiment Design Checker — run it, fix the holes it names, run it again.
• Don’t invent a benchmark when an established one exists; you’ll only make yourself incomparable.
• Decide the metric and the significance test before you write the protocol. If you find yourself choosing them after imagining the results, stop.
• For environment/Overleaf, see Appendix A; for the toolkit, Appendix B; for any unfamiliar term, Appendix D.

What Mastery Looks Like

A reader who has never met you takes your document and your repo, and within an afternoon knows exactly what to run, against what, how many times, and how to decide whether you were right — and could do it. There are no holes for them to fall through. The baselines are fair, the ablations isolate the effect, the metric and test were locked before the data, and every number will be traceable to a seed, a config, and a commit. The design reads like a promise you intend to keep.

Coach’s Note — The temptation this week is to leave the design loose so you “keep your options open” for Week 11. That openness is exactly the door p-hacking walks through. Lock it now. A locked design is freedom, not constraint — it’s the freedom to trust your own results.

When You’re Done

• Every field of experiment-design.yaml is filled.
• Naive + strong baseline named; strong-baseline fair-tuning plan written.
• One ablation per claim in the baseline matrix.
• Benchmark version-pinned; contamination/construct threat addressed.
• Splits defined; leakage guard stated; test-set rule written.
• Primary metric + direction; ≥ 5 seeds; significance test + effect size chosen in advance.
• Reproducibility checklist ≥ ~0.8; env/seeds/data/commit/LLM pinning/disclosure in place.
• A classmate could rerun your plan without asking you a question.
• Committed and pushed; commit hash submitted.

A theological footnote. Luke did not ask Theophilus to take his word for it. He wrote an orderly account of things followed closely, so that another could have certainty (Luke 1:3–4, ESV). That is what a design document is: an orderly account written so a stranger can check you and arrive at the same place. To design for reproduction is to refuse the temptation to be merely believed, and to choose instead to be examined — “the one who states his case first seems right, until the other comes and examines him” (Prov 18:17, ESV). You are making it possible for the next researcher to know what you know, instead of asking them to trust that you know it. In a small, concrete, seeded-and-committed way, that is loving your neighbor — and bearing true witness to the truth you searched out.