Project 12 — Results Section Draft

“A false balance is an abomination to the LORD, but a just weight is his delight.” — Proverbs 11:1 (ESV)

Chapter: Chapter 12 — Analyzing the Data Due: end of Week 12 (see the course schedule) Submit: a Results section (PDF from your acmart/IEEEtran Overleaf draft) + the analysis notebook (.ipynb) + the corrected-comparison table, committed and tagged in your project Git repo Allowed tools: Python (NumPy / SciPy 1.18.0 / statsmodels / scikit-learn / deep-significance) or R; Jupyter; matplotlib/seaborn; your raw results dataset from Chapter 11 AI policy: Open-AI for analysis code and prose drafting — you verify every test’s assumptions and every number against the primary tool/source, and you log substantive AI use in agent-log.txt. The model accelerates; you decide, verify, and are accountable (Appendix C). An LLM cannot be an author and “the model did it” is never a defense.

This is not a graded-weight deliverable on its own — it is the analytical core that feeds the Experimental Results (already banked in P11, 15%) into the Draft Paper (Week 14, 10%) and Final Paper (P14, 15%). Get the statistics honest here and the rest of the paper inherits the integrity. Get them wrong here and no amount of good prose later will save it.

The Setup

You have a results file: numbers, with provenance, from runs that actually happened. A reviewer two months from now — and your advisor next week — will read your Results section and ask exactly two questions. How big is the effect, and how sure are you? If your section answers those with effect sizes and confidence intervals, you’ve written a results section. If it answers them with a wall of bare p-values and a bar chart with no error bars, you’ve written a sales brochure, and a good reviewer will say so in three sentences.

Your job this week: turn the raw record into the honest, defensible Results section of your paper. Not the Discussion (that’s Chapter 13) — no interpretation, no “this suggests,” no story. Just: here is what we measured, here is how big, here is how sure, here is the test, here is the figure. The just weight, on the page.

Learning Targets

You will demonstrate that you can:

• Report effect size + confidence interval + the right p-value together, never one alone.
• Choose a statistical test that fits your data’s shape (paired/unpaired, normal/skewed, multi-seed) and justify it.
• Correct for every comparison you ran (FWER or FDR) and state the count of tests.
• Compare model variants across multiple seeds (Bouthillier, 2021) using an appropriate multi-seed test where applicable.
• Produce figures that show uncertainty, use defensible axes, and have self-contained captions.
• Write Results prose that reports without overclaiming — the line you’ll defend in the Discussion.

Normal Tier

Meets the deliverable bar for the week: an honest, reproducible Results section for your primary claim.

Required features

• Primary comparison reported with all three numbers: effect size (e.g., Cohen’s d or accuracy delta), 95% confidence interval, and the p-value from an appropriate test — in that order of emphasis.
• Test choice justified in one or two sentences (assumptions checked; permutation/non-parametric used where assumptions fail).
• At least one figure with uncertainty shown (CI/error bars/raw points), a defensible axis range, and a stand-alone caption.
• A reproducible analysis notebook that regenerates every number from the raw data on pinned library versions (Restart-and-Run-All clean).
• A multiple-comparison statement: how many comparisons you ran and which correction you applied.
• Plain Results prose — measurements only, no interpretation.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Stronger rigor and breadth:

• Multi-seed analysis of your headline comparison (≥5 seeds) with a distribution-level test — deepsig.aso (ASO, τ = 0.2) for DL scores, or a permutation/bootstrap analog — and a figure showing the seed spread.
• A full corrected-comparison table (every metric × dataset × subgroup) with both an FWER (holm) and an FDR (fdr_bh) column, so the reader sees what survives each.
• An error-analysis subsection that stratifies the primary metric by a meaningful slice and surfaces a systematic failure the aggregate hid.
• Confirmatory/exploratory split clearly labeled, so post-hoc findings aren’t dressed as predictions.

Hard Tier (+up to 25% additional extra credit)

Publication-ready, and demanding the judgment a tool can’t supply.

Do everything above, plus write a one-page Statistical Integrity Memo (a decision an AI cannot make for you) that argues, in your own voice:

1. The honesty audit. Identify the single result in your own data most vulnerable to a p-hacking or HARKing critique, name why, and state what you did about it (dropped it, labeled it exploratory, preregistered the re-test, reported it with the count). Show the count of comparisons that produced it.
2. The practical-vs-statistical call. For your headline result, argue whether the effect is practically meaningful in your domain — not just statistically significant — citing field-calibrated interpretation (Cohen’s benchmarks were arbitrary; Ortloff et al., CHI 2025). Commit to a defensible position.
3. Preregistration plan. State the one confirmatory analysis you will preregister on OSF before the full Discussion, and why locking it now protects the claim.

The memo is graded on the quality of judgment, not length. A confident, well-argued “this effect, though significant, is too small to matter, and here’s why I’m reporting it anyway” beats a page of hedging.

Submission

1. Results section as a PDF from your Overleaf acmart/IEEEtran draft (the same template you started in Week 1).
2. The analysis notebook (.ipynb), reproducible top-to-bottom on pinned versions.
3. The corrected-comparison table (CSV or in the notebook).
4. (Hard) integrity-memo.txt.
5. agent-log.txt if you used AI substantively.
6. All committed and tagged (e.g., git tag results-draft-v1) in your project repo.

Hints

• Start from code/analyze_results.py — it wires effect size, bootstrap CI, permutation test, and multipletests together.
• Use code/results_section_template.txt for the section skeleton and the three-number sentence pattern.
• Lead every result sentence with the effect, not the p. If a sentence’s only number is a p-value, rewrite it.
• Count your comparisons honestly — include the ones you ran and didn’t report. The denominator is part of the result.
• Restart Kernel and Run All before you submit. Out-of-order cells are how unreproducible numbers escape.
• Don’t interpret yet. “Method A scored 4.1 points higher (95% CI [1.2, 7.0])” is Results. “Method A is better because…” is Discussion — save it for Chapter 13.

What Mastery Looks Like

A reader who sees only your effect sizes and confidence intervals — never your p-values — can still judge whether your contribution matters. Your figures show uncertainty so plainly that no one has to ask “how many seeds?” Your comparison count is stated, your correction is applied, and the result that didn’t survive correction is in the table anyway. Nothing in the section overclaims; every sentence is a measurement you’d defend under oath. The Discussion will have plenty to interpret — but it will never have to walk back a number.

Coach’s Note — The Results section is where you’re most tempted to let the prose carry weight the statistics won’t support. Don’t. Keep Results boring and true. Save the meaning for the Discussion, where you can argue it honestly — after the reader already trusts your numbers.

When You’re Done

• Primary claim has effect size + CI + p-value, in that order of emphasis.
• Test choice is justified against the data’s shape.
• Every comparison counted; correction (Holm and/or BH) applied and stated.
• At least one figure shows uncertainty, defensible axis, stand-alone caption.
• Notebook reproduces every number on a clean Restart-and-Run-All with pinned versions.
• Results prose reports — it does not interpret.
• (Medium/Hard) multi-seed test, full corrected table, error analysis, integrity memo.
• Committed and tagged; agent-log.txt present if AI was used substantively.

A theological footnote. The merchant in Proverbs 11:1 kept two weights — a real, physical, accurate weight for each transaction — and was still an abomination, because the correspondence between his number and the truth was false. P-hacking is the scholar’s two weights: every number real, the whole report a lie. To weigh justly is to report the comparison you ran and not the one you fished for; to carry the effect size that embarrasses you; to keep the run that didn’t work. You do this not because a reviewer is watching — most of the time none is — but because the just weight is His delight. You are the steward of the data, not its customer. Weigh it as before the One who already knows what it says.