Project 11 — Raw Results Dataset

“You shall not bear false witness against your neighbor.” — Exodus 20:16 (ESV)

Chapter: Chapter 11 — Full Experimental Execution Due: End of Week 11 (see course calendar) Submit: A versioned link (Git tag + Zenodo DOI if available) to your repository containing the raw results dataset, the run log, the experiment journal, the environment lock, and a one-page integrity statement. PDF the integrity statement to Canvas. Allowed tools: Full research environment. AI may help write harness/logging/analysis-prep code and explain errors. AI may not generate, edit, alter, or “clean up” results. If an LLM is your experimental instrument, disclose and log it per Appendix C. AI policy: This is open-tool for the engineering, closed for the data. The model can help you build the ledger; the model may never touch what goes in it. Weight: 15% of your course grade (the “Experimental Results” deliverable).

The Setup

It is Week 11. Eight weeks ago you stood in front of the class and proposed this work. Seven weeks ago you wrote a design document another researcher could rerun. Last week your pilot proved the pipeline runs. The planning is over.

This week you run the real experiment — your full matrix, every seed, every baseline, every ablation — and you produce the single artifact that the rest of your paper stands on: the raw results dataset. Chapter 12 will run statistics on it. Chapter 13 will interpret it. Chapter 14 will write it up. If this dataset is incomplete, untraceable, or quietly curated, everything built on top of it is compromised — and a reviewer who can smell a dropped seed will say so.

Here is the scenario that makes the work matter. Imagine a reviewer, six months from now, reading your submission and asking the one question that ends careers: “How was this number produced?” Your dataset is your answer. If every row carries its commit, seed, config, and dataset hash — and your LLM calls carry their version, date, and prompt — you answer in one sentence and the reviewer moves on. If it doesn’t, you don’t have a finding. You have a story you can’t back up.

You are not graded on whether your method won. You are graded on whether you bore true witness to what happened.

Learning Targets

You will demonstrate that you can:

• Execute a full experimental matrix (all seeds, baselines, ablations) under a budget you computed in advance.
• Capture per-run provenance automatically — commit, seed, config, dataset hash — from inside the harness.
• Capture AI-instrument provenance where applicable — model version, date, verbatim prompt, sampling params — and verify model outputs before they become data.
• Keep an append-only, present-tense run log and experiment journal recording anomalies as they occur.
• Distinguish a documented mechanical exclusion from the temptation to drop a disappointing result, and act with integrity on both.
• Pin and archive a reproducible environment so a stranger could rerun any number you report.

Normal Tier

The bar for the week: a complete, honest, fully-provenanced raw dataset.

Required features

1. Full execution. Every cell your design document and proposal promised: all planned seeds × all conditions (your method, at least one baseline, at least one ablation per claim) × the official splits. No silent gaps.
2. Per-row provenance. Every result row carries: commit (clean tree — no dirty=true), seed, full config, dataset_sha256 + split, metric_name, metric_value. Provenance is written by the harness, not by hand. (Use code/provenance.py; code/run_log_template.csv is the schema.)
3. Run log + experiment journal. The numbers in a machine-appended CSV; the narrative in a present-tense journal (code/experiment_journal.txt) recording at least one anomaly noted in the moment, all exclusions with mechanical reasons + evidence, and any kept-despite-disliking results.
4. AI-instrument log (if applicable). Per-call model version string, provider, date/time, verbatim prompt (or hash), and temperature; plus evidence you verified a sample of model outputs against ground truth before trusting them.
5. Environment + archival. A pinned environment (environment.yml or requirements.txt), a tagged Git release, and a README paragraph stating how to rerun.
6. Integrity statement (one page). Plain-language: what you ran, how many runs landed, what you excluded and why, what you kept despite disliking it, and whether a stranger could reproduce any number you report. Complete code/repro_checklist.txt and attach it.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Stronger rigor and a cleaner reproducibility chain:

• Automated provenance gate: your harness refuses to launch from a dirty tree (or hard-fails the row), so a dirty=true result is structurally impossible.
• Experiment tracking: wire an experiment-tracking tool that auto-captures metrics, params, code version, and environment per run, and link it from your README.
• Zenodo DOI: deposit the tagged release to Zenodo, mint a DOI, and confirm the auto-linked Software Heritage SWHID. Cite the DOI in your integrity statement.
• Variance reporting readiness: present your raw per-seed spread (not aggregated — that’s Chapter 12), demonstrating you have enough seeds to say something statistically real (Bouthillier et al. 2021).

Hard Tier (+up to 25% additional extra credit)

Publication-grade — and it demands a judgment a tool cannot make for you.

Write a 2-page “Integrity & Threats to the Record” memo that an AI cannot write for you, because it requires honest self-examination of your runs. In it:

• Identify the one result in your own dataset you were most tempted to drop, spin, or quietly rerun until it behaved — and explain, specifically, why you kept it and what it costs your hypothesis.
• Map every exclusion to its mechanical evidence and argue why each is a defensible falsification-free exclusion under the 42 CFR Part 93 definition (which includes omitting results).
• If an LLM is your instrument: assess where its drift or fabrication (arXiv:2603.08274) most threatens the validity of your record, and what you did to bound that threat.
• Close with the single biggest threat to anyone reproducing your raw results, and your concrete mitigation.

The memo is graded on intellectual honesty, not on how good your numbers look. The strongest memos admit the most.

Submission

1. Push your repo with the raw dataset, run log, journal, environment lock, and README.
2. Tag the release (git tag -a results-v1 -m "..."); include the tag in your submission.
3. (Medium+) Deposit to Zenodo; include the DOI.
4. PDF the one-page integrity statement (+ Hard memo, if attempting) to Canvas.

Hints

• Build the logging before the first launch. Retrofitting provenance after the runs is the classic Week-11 disaster (§11.6).
• Set seeds everywhere and commit before every run. These two habits remove 90% of “how did I get this number?” panic.
• Don’t aggregate. This week is the raw record — per-run rows, not means. Means and tests are Chapter 12.
• For LLM instruments, temperature 0 is not determinism — log it and run multiple seeds anyway (§11.3).
• When a number disappoints you, that’s the moment to slow down and write it carefully, not delete it quickly.

What Mastery Looks Like

A stranger clones your repo, reads the README, reruns one cell, and gets your number — and when they scan your log they find the runs you could have hidden, documented in plain sight. Your integrity statement is specific enough that a skeptical reviewer relaxes. Your method may not have won every seed. It doesn’t matter. The record is true, and it is complete, and it is yours to stand behind.

Coach’s Note — The single highest-leverage thing you can do this week is decide, before you launch, that you will report whatever comes back. Make that decision once, in advance, and the hundred small temptations of the week lose their grip. You can’t be tempted to hide a result you already committed to reporting.

When You’re Done

• Full matrix executed; row count = conditions × seeds (minus documented exclusions).
• No dirty=true rows; every row has commit/seed/config/data hash.
• Journal records anomalies in the moment; exclusions have mechanical reasons + evidence.
• LLM calls (if any) carry version/date/prompt/temp; outputs verified.
• Environment pinned; release tagged; README says how to rerun.
• One-page integrity statement written, true, and matching the dataset.

A theological footnote. The eighth commandment — “You shall not bear false witness against your neighbor” (Exodus 20:16, ESV) — is usually read as a courtroom rule, and Luther’s Small Catechism widens it: not only to refrain from lying about our neighbor but to “defend him… and explain everything in the kindest way.” Your dataset is a kind of neighbor — a voiceless witness whose testimony you relay to everyone who will ever read your paper. To drop the inconvenient seed is to put words in that witness’s mouth. This project is the discipline that makes faithful witness possible: a record so complete that you could not lie to your reader without it showing. You do this work coram Deo — before the face of God, who sees seed 3 whether your reviewer does or not — and there is a real freedom in it. The researcher who has decided to report whatever comes back is free in the lab in a way the cherry-picker never is. “The truth will set you free” (John 8:32, ESV) turns out to be excellent experimental advice.