Chapter 15 — Peer Review and Revision

“Reviewing is the immune system of science. It is slow, it is imperfect, and the field would die without it.” — paraphrased from the research-methods literature on peer review

“Whoever loves discipline loves knowledge, but he who hates reproof is stupid.” — Proverbs 12:1 (ESV)

Why This Matters

Last week you wrote a paper. This week the field writes back.

Peer review is the moment your work stops being yours and becomes a claim — something other competent people are allowed to test, doubt, and push on. Every paper you have read this semester in the ACM Digital Library, on IEEE Xplore, on OpenReview, survived this. The method section you trust, the result you cite, the baseline you copied — all of it cleared reviewers who were paid nothing and given no benefit of the doubt. That is the immune system. It is the reason a published result means more than a blog post, and it is the reason your draft is not finished until it has been through it.

This is a drill week. No new project file. You will do two things, and they are the whole grade: you will formally review two of your peers’ papers, and you will revise your own paper in response to the reviews it receives. Both halves are hard, and they are hard in opposite directions. Reviewing demands that you be specific and fair about someone else’s work without being cruel or lazy. Being reviewed demands that you read sharp criticism of your own work without flinching, without sulking, and without ignoring it. Most researchers are bad at exactly one of these. The good ones train both.

The AI thread runs straight through this week from both sides. As a tool, an LLM will happily draft you a review — and it will hallucinate weaknesses, miss the real one, and, if you paste a confidential manuscript into a public chatbot, breach the very policies the major venues now enforce against reviewers. As a workload, the papers you are reviewing are about AI in your domain — so part of reviewing well is knowing the failure modes of the method under test (a single-seed “we beat the baseline,” a contaminated benchmark, a hallucinated citation in the related-work section). The human stays where the judgment lives. The reviewer decides. The author is accountable. No model can hold either role.

And here is the apologetic question this week, posed plainly: what does it mean to love correction? Proverbs does not say the wise tolerate reproof or survive it. It says they love it — (Prov 12:1, ESV) — because correction is how a finite mind gets closer to the truth it does not yet have. That is a strange thing to love. We will come back to why a researcher, of all people, should.

Coach’s Note — The single best predictor of whether a student’s paper ever gets published is not how good the first draft was. It’s how they took the first round of reviews. I have watched brilliant drafts die because the author argued with every reviewer, and I have watched mediocre drafts become real papers because the author treated three harsh reviews as a free roadmap. You are about to find out which kind you are. Choose.

15.1 — How Peer Review Actually Works

Strip away the mystique. Peer review is a structured process with a small number of moving parts:

1. Submission — you upload an anonymized manuscript to a venue’s review system before a hard deadline.
2. Assignment — an Area Chair / Senior PC member assigns it to 3–5 reviewers with matching expertise.
3. Reviewing — each reviewer reads independently and writes a structured review: summary, strengths, weaknesses, questions, scores, and a recommendation.
4. Rebuttal / discussion — authors respond to the reviews; reviewers and the chair discuss; scores may move.
5. Decision — accept / reject (with conditions, sometimes), made by the chair on the basis of the reviews and discussion.
6. Camera-ready — accepted papers are de-anonymized and finalized.

Most top CS venues review double-blind (the modern term is double-anonymous): the authors don’t know who the reviewers are, and the reviewers don’t know who the authors are. De-anonymization happens only at camera-ready. The point is to judge the work, not the name — a result from an unknown student and a result from a famous lab get read the same way.

The systems differ by venue. As of mid-2026:

Coach’s Note — Go read a real OpenReview thread before you write a single review of your own. Open an ICLR or NeurIPS submission, read the reviews, read the rebuttal, read the discussion. You are watching professionals do the exact thing you’re about to do. Notice that the good reviews are specific — they point at a figure, a line, an equation — and the bad ones are vague (“the contribution seems incremental”). Be the first kind.

15.2 — Submitting for Review: The Anonymity Discipline

Before anyone reviews your work, you have to hand it over in the right shape. Double-anonymous review means you are responsible for scrubbing yourself out of your own paper. A paper that reveals its authors can be desk-rejected without review.

The acmart class makes this mechanical: you submit with the anonymous option, which prints “ANONYMOUS AUTHOR(S)” in place of your byline.

% submission, double-blind:
\documentclass[manuscript,review,anonymous]{acmart}
% camera-ready, after acceptance:
\documentclass[sigconf]{acmart}

But the tooling only handles the byline. You handle the rest:

• Self-citations in the third person. Not “in our prior work [4],” but “prior work [4] showed.” The first phrasing tells the reviewer exactly which lab wrote this.
• Empty the Acknowledgements. It names your advisor, your lab, your funder.
• Strip grant IDs. A grant number identifies the institution as surely as a logo.
• Anonymize links. A raw github.com/yourname link de-anonymizes you instantly. Use an anonymized mirror (e.g. anonymous.4open.science) or an anonymized OSF view.
• Clear PDF metadata. The Author field in the document properties is the classic leak.

The full list is in code/anonymity-checklist.txt — run it before you submit your paper for the lab this week. De-anonymizing yourself is rarely cheating; it’s usually sloppiness. The outcome is the same: reviewers can’t unsee it.

15.3 — What Reviewers Actually Look For

A review is not a book report. It is a judgment along a small set of axes, and you owe evidence for each one. The five that matter most in applied-CS venues:

Notice what’s not on the list: whether you would have done it that way, whether you like the authors’ writing style, whether the topic is trendy. Reviewing your taste instead of the work is the most common rookie failure.

Because your peers’ papers are all about AI in a domain, three weaknesses recur often enough to name explicitly:

• The single-seed result. Deep-learning numbers swing with the random seed (Bouthillier et al. 2021, “Accounting for Variance in ML Benchmarks,” MLSys; arXiv:2103.03098). “We beat the baseline” from one run is statistically empty. Flag it.
• The contaminated benchmark. If the model was trained on data that overlaps the test set, the score is fiction. Contamination-resistant benchmarks (LiveCodeBench, LiveBench, MMLU-Pro) exist precisely because this is rampant.
• The hallucinated citation. LLM-assisted writing can invent references that look real and aren’t (Walters & Wilder 2023, Scientific Reports: ~55% of GPT-3.5 and ~18% of GPT-4 citations entirely fabricated). If you can’t find a cited paper in dblp or Semantic Scholar, that is an integrity flag, not a nitpick. The script code/verify_citations.py automates the lookup; the judgment about what a missing match means is yours.

15.4 — Writing a Constructive Review

There is a craft to this, and it has a shape. A strong review has the same parts whether it’s for NeurIPS or for your classmate:

1. Summary        — restate the paper's problem/method/result so the AUTHORS would agree.
2. Strengths      — specific, cited to section/figure. (Yes, even on a weak paper.)
3. Weaknesses     — tagged by severity: MAJOR / MINOR / CLARIFY.
4. Questions      — things a rebuttal could actually resolve.
5. Scores         — per axis, each justified in one line.
6. Recommendation — and the ONE change that would most move your score.

The full template is in code/review-template.txt. Three rules turn a mediocre review into a useful one:

Be specific, not vague. “The evaluation is weak” helps no one. “The evaluation reports a single run; please add seeds and 95% CIs (Table 2)” is a gift — it tells the author exactly what to do.

Tag severity honestly. A [MAJOR] weakness threatens a central claim. A [MINOR] is fixable and doesn’t sink the paper. A [CLARIFY] means you genuinely could not tell — which is sometimes the author’s fault and sometimes yours. Don’t inflate a typo to a major weakness, and don’t bury a fatal flaw in a list of typos.

Lead with the summary, and make it fair. If you can’t restate the paper in three sentences the authors would nod at, you don’t understand it well enough to judge it. Write the summary first. It will calm down a review you were about to write angry.

Coach’s Note — The civility rule isn’t politeness for its own sake. A cruel review gets discarded — the author’s defenses go up, the chair discounts the tone, and your real point dies with the insult wrapped around it. Write the review you would want to receive: hard on the work, easy on the person. “This claim is not supported by Table 3” lands. “The authors clearly don’t understand statistics” does not.

15.5 — Using AI in Reviewing — and Where the Line Is

You will be tempted to paste your peer’s paper into a chatbot and ask for a review. Do not. As of mid-2026 the major venues have explicit reviewer-side AI policies, and they are among the most volatile, year-specific rules in this whole book — re-check your target venue’s current page (see Appendix C). The consensus as of mid-2026:

The common thread: the manuscript is confidential, and the judgment must be yours. Pasting a peer’s unpublished paper into a public chatbot is a confidentiality breach regardless of how good the review comes out. And a review the model wrote is a review you didn’t do — it will sound fluent and miss the real weakness, because the real weakness usually lives in the gap between what the paper claims and what its evidence supports, which is exactly the judgment a tool can’t make and you can’t outsource.

For your own paper, AI assistance is governed by the venue’s author-side disclosure policy (also in Appendix C): assistive editing is generally exempt; substantively generated text, figures, code, or ideas get disclosed — typically in the Acknowledgements (ACM, IEEE, ACL) or experimental setup (NeurIPS), with the system named. “The model did it” is never a defense. The human author owns every word, figure, citation, and line of code.

15.6 — Reading Reviews of Your Own Work

Now the hard half. Your reviews come back. Some are fair, some are wrong, one is curt. Here is the discipline, in order:

1. Read them all once. Then close the file and do something else. The first read triggers the defensive reflex — “the reviewer didn’t even understand it.” That reflex is useless. Let it pass.
2. Read them again, and convert each comment into a task. Strip the tone. A harsh sentence and a kind sentence asking for the same thing are the same task. “Did the authors even run more than one seed?” and “Please add multi-seed results” both mean: add seeds.
3. Sort every comment as Done, Will-do, Disagree, or Clarify. Most are Done or Will-do. A few you’ll respectfully disagree with — and that’s allowed, if you can show evidence. “Clarify” means the reviewer misread something, which usually means your writing let them.
4. Find the agreement across reviewers. When two reviewers independently flag the same thing, it is real. Fix it first, no matter how it stings.

The hardest reviews to take are the ones that are right. A reviewer catches a leakage bug, a missing baseline, an overclaim you half-knew was there. That is the review doing its job — it found the flaw before the published record did. Fixing it is not a humiliation; it is the entire point of the process.

Coach’s Note — I keep a private file of the worst review I ever got. It opened: “The authors appear unaware of an entire literature.” It was right. That paper, after I read the twenty papers the reviewer was implicitly pointing at, became the best thing I’d written. A reviewer who tells you you’re wrong is doing you a larger favor than a reviewer who tells you you’re great. The second one costs you nothing and teaches you nothing.

15.7 — Revising and Writing the Rebuttal

The rebuttal (or “author response”) is your structured reply to the reviews. It is not an argument — it is a point-by-point account of what you changed and why. Done well, it moves scores. Done badly, it confirms every reviewer’s worst read of you.

Venues cap it tightly. As of mid-2026, NeurIPS allows 10,000 characters per review, plain text plus OpenReview markdown, no file uploads (only an anonymized code link if a reviewer asked for one). New results are allowed during rebuttal, but the original submission remains the basis for the decision — so promise only the changes you can actually make in time. Per-venue character and page limits change every year; verify the exact number for your target.

The shape is in code/rebuttal-skeleton.txt: a short global response naming the 2–3 biggest changes, then a per-reviewer, per-comment reply with a stable tag (R1.1, R1.2…). Three rules:

• Lead with the verb. Start each response with Done / Fixed / Clarified / Added / Respectfully clarified. The reviewer should know in the first word whether you acted.
• Never claim a change you can’t make, and never concede a result you didn’t change. A rebuttal that over-promises reads as desperate; one that quietly re-states the unchanged paper reads as stubborn.
• When a reviewer is simply wrong, show the evidence calmly. Don’t get the last word. Get the right one. “We respectfully clarify: the scaler is fit on the training fold only (§4.1), so the leakage the reviewer describes does not occur.” Calm, specific, done.

A worked example, the kind of exchange this whole week is training:

> R1.1 (MAJOR): "Results are single-seed; the gain may be noise."
Response — Done. We reran across 10 seeds (Table 3, revised). Mean gain +2.4 points
(95% CI [1.1, 3.7]); an Almost Stochastic Order test (deep-significance, tau=0.2)
confirms dominance over the baseline. Text updated in section 5.1.

That is a reviewer’s objection, an author’s honest fix, and the statistics done right — the entire arc of the course in four lines.

15.8 — Interactive Lab: Peer-Review Workshop

Open the Peer-Review Workshop widget embedded on this page, below the chapter.

You’ll work through three drills that rehearse exactly what the graded lab demands:

1. Classify the comment. The widget shows you real-style review comments; you tag each as MAJOR, MINOR, or CLARIFY. It teaches you to feel the difference between a flaw that sinks a paper and a typo — the calibration every reviewer needs and most students lack.
2. Rewrite the harsh one. It hands you a cruel, vague, or condescending comment (“the authors clearly don’t understand statistics”) and asks you to rewrite it as a specific, constructive, citable note. This is the single most valuable reviewing skill, and you can’t fake it.
3. Draft the point-by-point response. It gives you a review and asks you to draft a tagged, verb-first rebuttal entry under a character budget — the same constraint NeurIPS imposes.

Do all three before you start the lab in the exercises. The widget is a flight simulator; the lab is the flight.

15.9 — On Loving Correction

What does it mean to love correction?

It is one of the strangest things Scripture asks of a mind. “Whoever loves discipline loves knowledge, but he who hates reproof is stupid” (Prov 12:1, ESV). The verse is blunt on purpose. It does not flatter. It says that the disposition toward correction is the disposition toward knowledge — that you cannot have one while refusing the other — and it puts the refusal in a word, stupid, that no one wants to wear.

Here is why this lands harder on a researcher than on almost anyone else. Your whole vocation is the claim that you do not already know — that the gap is real, that the answer must be searched out (Prov 25:2, the verse this entire book is built on). A researcher who cannot take correction is claiming, in the same breath, both that they are searching for truth they don’t have and that no one may tell them where they’re wrong. That is incoherent. The reviewer who finds your leakage bug, the classmate who notices your single-seed result, the verse that calls the hatred of reproof stupid — they are all doing the same thing the scientific method does: subjecting your claim to a witness outside yourself. “Every charge must be established by the evidence of two or three witnesses” (2 Cor 13:1, ESV), which is to say: not by your own confidence in yourself.

There is a deeper register, and it is worth naming once plainly because the LCMS confession runs through this whole book. The Christian tradition has a word for the willingness to hear that you are wrong and turn: repentance. Peer review is not repentance — let’s not overclaim, in a chapter on overclaiming. But it rehearses the same humility. It assumes you are finite, that your sight is partial — “for now we see in a mirror dimly” (1 Cor 13:12, ESV) — and that correction from outside is a gift, not an attack. The proud author and the proud sinner make the identical mistake: they treat the witness who would correct them as an enemy, when in fact that witness is the only road to the truth they say they want. To love correction is to believe, in your bones, that the truth is bigger than your draft of it. A researcher who believes that will write the rebuttal with gratitude instead of resentment — and will write a better paper for it.

So when the reviews come back this week and the defensive heat rises in your chest, remember what you are looking at. Not an attack. A gift you asked for the moment you decided to search a matter out.

15.10 — Common Pitfalls

Pitfall: The vague review. Example: Your entire weakness section reads “the contribution is incremental and the evaluation could be stronger.” Fix: Cite the evidence. Which claim is incremental relative to which prior paper? What would make the evaluation stronger — seeds, a baseline, a CI? A weakness the author can’t act on is a weakness you didn’t really find.

Pitfall: Reviewing your taste instead of the work. Example: You downgrade a paper because you’d have used PyTorch, not JAX, or because the topic isn’t the one you’d have picked. Fix: Judge the paper on its own axes — novelty, method, honesty, clarity, significance. “I would have done it differently” is not a weakness. “This choice threatens validity because X” is.

Pitfall: Pasting a confidential manuscript into a public chatbot. Example: You drop your peer’s unpublished paper into a public LLM and ask it to “write a review.” Fix: Don’t. As of mid-2026 this breaches NeurIPS/IEEE/CVPR reviewer policies regardless of output quality, and the model misses the real weakness anyway. Use AI, if at all, only within the venue’s stated limits and never on the confidential text. See Appendix C.

Pitfall: Arguing with the reviewer instead of answering them. Example: Your rebuttal explains, at length, why the reviewer is too dim to appreciate your method. Fix: Convert every comment to a task. Lead with a verb — Done, Fixed, Clarified. If you disagree, show evidence calmly and move on. The chair is reading the tone as data.

Pitfall: De-anonymizing yourself. Example: “In our prior work [4]…”, a live github.com/yourname link, your name in the PDF metadata. Fix: Run code/anonymity-checklist.txt before submitting. Third-person self-citations, anonymized links, cleared metadata, emptied acknowledgements.

Pitfall: Conceding a fix you can’t actually make in time. Example: “We will add a full ablation across six datasets” — which you cannot finish before the deadline. Fix: Promise only what you can deliver. A rebuttal that over-promises and under-delivers is worse than one that honestly says “we ran the ablation on the two datasets we had compute for (Table 4); the remaining four are future work.”

Pitfall: Treating a hallucinated citation as a nitpick. Example: You notice a cited paper you can’t find anywhere and let it slide as “probably a typo.” Fix: A reference that doesn’t exist is an integrity flag, not a typo. Some venues (e.g. ICCV as of mid-2026) reject papers with non-existent citations without review. Flag it; run code/verify_citations.py on your own .bib before you ever ship.

15.11 — Reps

The reps live in the exercises, and this week they are the graded lab — there is no separate project file. You will formally review two peers’ papers and revise your own in response to the reviews you receive. A preview:

• Rep 1 — Run the anonymity checklist on your own paper and fix every leak.
• Rep 3 — Dissect one real OpenReview thread end to end and tabulate the reviewer objections.
• Rep 5 — Write the full formal review of Peer A’s paper on the template.
• Rep 7 — Triage the reviews you received: tag every comment by severity and convert each to a task.
• Rep 9 — Write the point-by-point response to your reviewers, inside the venue’s character budget.

Don’t skip the on-page Check Your Reps quiz below the widget — five questions straight from this chapter.

15.12 — This Week’s Deliverable

There is no project file this week. The lab in the exercises is the deliverable, and it is graded: two complete formal reviews of your peers’ papers, plus a revised version of your own paper with a point-by-point response to the reviews you received. The required tools are the review template, the anonymity checklist, the rebuttal skeleton, and the citation-verification script in code/.

This is the last gate before the symposium. Next week — Chapter 16 — you present the finished work and build a real publication roadmap toward an actual venue. The revision you do this week is the version you will stand behind. Make it the one the reviews earned.

15.13 — Coach’s Final Word

Reviewing is the job no one trains you for and everyone judges you on. You will do it, unpaid and anonymous, for the rest of your research life — and the field runs on whether you do it well. This week is your first real rep at both ends of it: holding the scalpel, and lying on the table.

The author who survives is not the one with the thickest skin. It’s the one who learned to read three hard reviews and hear, underneath the sting, a free map to a better paper. Proverbs got there three thousand years before peer review existed: the one who loves reproof is on the road to knowledge, and the one who hates it is going nowhere with great confidence. Be the first kind. Read the reviews twice, convert them to tasks, fix what’s true, and answer the rest with a verb and the evidence.

Then come back next week and defend the thing you built.

See you on Monday.

Up next: the graded lab in the exercises — review two peers, revise your own. Then Chapter 16, the research symposium and your publication roadmap. Reference: Appendix B (the researcher’s toolkit), Appendix C (using AI responsibly), Appendix D (glossary).