Chapter 15 — Reps (Graded Lab: Peer Review and Revision)

This week the reps are the deliverable. There is no separate project file. By Sunday you will have produced two complete formal reviews of your peers’ papers and a revised version of your own paper with a point-by-point response. This is the last gate before the symposium.

“Whoever loves discipline loves knowledge, but he who hates reproof is stupid.” — Proverbs 12:1 (ESV)

Ground rules

• Tools allowed. The code/ files (review-template.txt, anonymity-checklist.txt, rebuttal-skeleton.txt, verify_citations.py), your reference databases (dblp, Semantic Scholar, ACM DL, IEEE Xplore), Zotero, and Overleaf.
• AI policy. You may not paste a peer’s confidential manuscript into any public LLM — that breaches the reviewer-side policies the major venues enforce (see Appendix C). You may use AI to understand a concept in a paper, the way a NeurIPS reviewer is permitted to. For your own paper’s revision, AI assistance is governed by your target venue’s author-side disclosure policy; disclose substantive generative use. The judgment — what’s a major weakness, what to fix, what to push back on — is yours and is graded.
• Integrity. Every reference you cite (and every reference you flag in a peer’s paper) gets verified against a real index. No hallucinated citations, on either side of the review.
• Portfolio. Commit your two reviews, your revised paper, and your rebuttal to your research Git repository. The reviews you write are part of your portfolio, not just the paper you submit.
• Anonymity. You review double-blind. Don’t try to learn who wrote the paper; don’t reveal who you are in your review.

Pair-up logistics (who reviews whom) come from the course roster — assume you’ve been assigned Peer A and Peer B, and that two classmates have been assigned your paper.

Theme 1 — Prepare your own paper for review

Rep 1 — Run the anonymity checklist

Open code/anonymity-checklist.txt and run every author-side item against your own paper.

% flip your acmart submission to anonymous review mode:
\documentclass[manuscript,review,anonymous]{acmart}

Fix every leak: third-person self-citations, emptied acknowledgements, stripped grant IDs, anonymized repo/data links, cleared PDF metadata.

Reflection (3–5 sentences): Which leak did you almost miss? Most students catch the byline and forget one of: self-citations, the acknowledgements, or the live GitHub link. Name yours and how a reviewer would have used it to guess your identity.

Rep 2 — Verify your own citations before anyone else does

Export your references to refs.bib and run the checker:

python code/verify_citations.py refs.bib

For every [FLAG], open the paper by hand in dblp or Semantic Scholar. Confirm it exists, the metadata is right, and you actually read enough of it to cite it. Remove or fix anything you can’t open.

Reflection: How many references flagged, and how many were real but had a metadata error (wrong year, wrong venue)? Tie your answer to the citation-hallucination literature (Walters & Wilder 2023: ~55% of GPT-3.5 / ~18% of GPT-4 citations entirely fabricated) — if you used any AI assistance while writing, did it introduce any of these?

Theme 2 — Learn the move by watching professionals

Rep 3 — Dissect a real OpenReview thread

Pick one submission to a recent ICLR or NeurIPS cycle on OpenReview. Read the reviews, the rebuttal, and the discussion end to end. Tabulate every reviewer objection of the form “the authors did not…”:

Reflection: Which objection moved the score most? Reviewers almost always weight methodology/honesty objections (missing baseline, no seeds, leakage) above writing nitpicks. Does your table confirm that, or did clarity matter more than you expected?

Rep 4 — Calibrate severity on the widget

Open the Peer-Review Workshop widget on the chapter page and complete all three drills: classify comments by severity, rewrite a harsh comment constructively, and draft a point-by-point response. Do this before you review a real peer.

Reflection: On the classify drill, where did your tagging disagree with the widget’s key? The line between MAJOR and MINOR is the hardest calibration in reviewing — describe one comment you mis-tagged and why.

Theme 3 — Review two peers (the heart of the lab)

Rep 5 — Full formal review of Peer A

Using code/review-template.txt, write a complete, structured review of Peer A’s paper: neutral summary, specific strengths, severity-tagged weaknesses, questions, per-axis scores, integrity flags, and a recommendation with the one change that would most move your score.

Run verify_citations.py mentally (or literally, if you have their .bib) on any reference that looks suspicious. A reference you can’t find in dblp/Semantic Scholar is an integrity flag, not a nitpick.

Reflection: What was the single most important weakness, and is it MAJOR? Write the one sentence the authors most need to read — specific enough to act on, civil enough to be heard.

Rep 6 — Full formal review of Peer B

Repeat for Peer B. Same template, same rigor. Do not let the second review get lazy — the second one is where students start writing “evaluation could be stronger” instead of naming the missing baseline.

Reflection: Compare the two papers’ biggest weakness. Were they the same kind of problem (e.g. both single-seed, both contaminated benchmarks, both overclaiming relative to the CIs)? If a pattern repeats across your cohort, name it — that pattern is the gap the whole field is fighting.

Theme 4 — Take your reviews and revise

Rep 7 — Triage the reviews you received

You now hold two reviews of your own paper. Read them once, close the file, walk away. Then read again and build the triage table — one row per comment:

Mark every comment that two reviewers raised independently — those are real, fix them first.

Reflection: Which comment stung the most, and was it right? The reviews that hurt are usually the ones that found something you half-knew. Name it honestly. (No one grades the sting; they grade whether you converted it to a task.)

Rep 8 — Revise the paper

Make the changes. Prioritize: agreed-upon MAJOR comments first, then MAJOR, then MINOR, then CLARIFY (which usually means rewriting so the reviewer wouldn’t have been confused). If a review asked for multi-seed results, add them with 95% CIs and a proper test (e.g. the Almost Stochastic Order test via deep-significance, τ=0.2) — don’t fake a fix with a single rerun.

Commit the revision to your Git repo with a message that references the reviews (“revise: multi-seed table + corrected baseline per reviews”).

Reflection: What did the revision cost you — a claim you had to soften, a result you had to qualify, a section you had to cut? Honest revision usually removes an overclaim. What did you stop saying?

Rep 9 — Write the point-by-point response

Using code/rebuttal-skeleton.txt, draft your rebuttal: a short global response naming your 2–3 biggest changes, then a per-reviewer, per-comment reply. Lead every entry with a verb (Done / Fixed / Clarified / Added / Respectfully clarified). Stay inside a realistic budget — NeurIPS allows 10,000 characters per review as of mid-2026, no file uploads (verify your target venue’s exact limit).

Reflection: Find the one comment you genuinely disagree with and write the “Respectfully clarified” entry. Did you show evidence and stop, or did you keep going to get the last word? Trim until it’s calm.

Rep 10 — The integrity pass

One final read of your revised paper with one question: does every sentence of claim point to a specific result that supports it? Mark any sentence where the claim outruns the evidence (the overclaim a reviewer will catch). Then check the AI-disclosure: if you used substantive generative assistance anywhere in the paper, is it disclosed where your target venue requires (Acknowledgements for ACM/IEEE/ACL; experimental setup for NeurIPS)? See Appendix C.

Reflection: List one claim you had to soften because the effect size or CI didn’t support the strong version. This is the discipline of Proverbs 11:1 — a just weight — applied to your own sentences.

Done? One Last Thing.

Put it together the way the symposium will demand. Assemble your revision packet: the revised paper (anonymized form for the record, plus your de-anonymized camera-ready-style version), the point-by-point response, and a one-paragraph reviewer’s reflection — what reviewing two papers taught you about your own. Almost every student, after reviewing peers, sees a flaw in their own paper they were blind to before. Name it, and confirm you fixed it.

Then write the two sentences that close the loop: the harshest true thing a reviewer said about my paper, and what I did about it. That is the whole week. If you can write those two sentences without flinching, you have learned to love correction — and you are ready to defend the work next week.

Commit everything to your portfolio repo.

Up next: Chapter 16 — the research symposium, the 12-minute talk, and your publication roadmap toward a real venue.