Project 11 — Find the Bugs in AI’s Code

“Test the spirits to see whether they are from God.” — 1 John 4:1

Chapter: 11 — Code Review Due: End of Week 11 Submit: A link to your code containing four bug-hunt-N/ subdirectories, your bug-log.txt, fixed/ versions, the prompt-templates document (for Medium), and prompts.txt. See Coding 1’s online-coding workflow appendix. Allowed tools: Java 17, JUnit 5, your AI assistant (for context, classification, or Medium tier), the Java 17 docs, your eyes. Required artifact: a documented bug-log.txt with category, line, evidence, and fix for every bug you find.

The Setup

This week you flip the polarity. For three weeks you’ve been producing AI code. This week you audit AI code that someone else (the course staff, posing as a fellow engineer) produced — with planted bugs in every file.

You will receive four small Java programs, each between 30 and 80 lines, each from a different problem domain, each containing at least two real bugs drawn from the six categories of Chapter 11. The bugs are the kinds AI actually makes: hallucinated APIs, plausible-but-wrong logic, null mishandling, off-by-one, performance traps, convention drift.

Your job:

1. Find every bug. The grader has a key. You don’t.
2. Classify each bug. Category from §11.2–§11.7.
3. Fix each bug. Cleanly, minimally — don’t refactor; just fix.
4. Write tests that would have caught each bug.
5. Document everything in bug-log.txt.

This is the most reading-heavy project in Phase 2. You will spend more time reading than typing. That ratio is what senior engineering actually feels like.

Learning Targets

By completing this project, you will demonstrate that you can:

• Read unfamiliar AI-generated Java code methodically.
• Identify bugs across all six categories from Chapter 11.
• Classify bugs accurately (the classification is part of the grade).
• Write JUnit tests that catch identified bugs.
• Produce minimal, correct fixes without unnecessary refactoring.
• Document a code review in a form a colleague or grader could trust.
• (Medium) Compose prompt templates that would have prevented categories of bugs in the first place.

Normal Tier

Goal: Find, classify, fix, and test every bug across four AI-generated programs.

Where to get the four programs

The four starter files are in the p11-starter/ folder, attached to this week’s release on the course portal. They are:

1. bug-hunt-1/CsvNormalizer.java — reads a list of strings, normalizes whitespace, returns the cleaned list. ~40 lines. Theme: data processing. (download CsvNormalizer.java)
2. bug-hunt-2/HistoryBuffer.java — a circular buffer that keeps the last N items added. ~50 lines. Theme: data structure. (download HistoryBuffer.java)
3. bug-hunt-3/WordFrequency.java — counts how often each word appears in a list of sentences, returns sorted (most frequent first). ~60 lines. Theme: collections. (download WordFrequency.java)
4. bug-hunt-4/DateRange.java — represents a date range with overlap, contains, and difference operations on LocalDate. ~70 lines. Theme: domain logic. (download DateRange.java)

Each file has a header comment with the spec the AI was given. The bugs are real AI bugs — the kinds we have seen models produce. Each file contains at least 2 bugs, possibly more.

Don’t hunt for an exact bug count. Hunt until you stop finding bugs. Then run your tests. Then re-read. The exercise is the discipline, not the count.

Required deliverables (per starter file)

For each of the four bug-hunt-N/ directories, you produce:

1. original.java — the starter file, unmodified, for reference.
2. fixed.java — your fixed version. Minimal changes — fix the bug, don’t redesign the program.
3. Tests.java — your JUnit 5 test class. At least 5 tests per starter, including at least one test per bug you found.
4. bug-log.txt — your findings for this file. Format:

# bug-hunt-N: <FileName>

## Bug 1
- **Category:** [hallucinated API / null-handling / off-by-one / plausible-but-wrong / performance / convention drift]
- **Location:** line(s) X–Y
- **Evidence:** [paste the exact line(s), or show the failing test output]
- **Why it's wrong:** [1–3 sentences]
- **Fix:** [paste the corrected line(s)]
- **Test that catches it:** [name of the test in Tests.java]

## Bug 2
[same structure]

## Bug 3
[same structure, if you found more]

## What I almost missed
[1 paragraph: a bug you only caught on second pass, and why]

Required deliverables (project-wide)

5. bug-log.txt at the project root — a concatenation or summary of all four per-file bug logs, plus a top-level summary:

# Project 11 — Bug Log

## Summary
- Total bugs found: N
- By category: [count per category]
- Average time per file: M minutes

[then the per-file logs follow]

6. prompts.txt — chronological log of every prompt you sent the AI this week. (Note: for Project 11, you might prompt the AI very little — most of the work is reading. Log what you did prompt for, even if it’s “I asked Claude to explain what LocalDate.until returns when the dates are reversed.”)

Grading rubric — Normal (out of 100)

Per-file scoring (18 points each) breaks down roughly:

• Found every planted bug: 8
• Classified each bug correctly: 4
• Tests catch each bug (before fix) and pass (after fix): 4
• bug-log.txt entry for this file is complete: 2

If you find more bugs than the planted ones, document them. Credit if they are real bugs; no penalty if they aren’t (though “no penalty if you’re wrong” is also your hint that not every suspicion is right — classify carefully).

Medium Tier (+up to 25% extra credit)

For each bug category you found at least one bug in, write a prompt template that would have prevented that bug class from being generated.

For example, if you found a hallucinated API bug, your prompt template might add:

“Use only methods documented in the Java 17 standard library at docs.oracle.com/en/java/javase/17/docs/api/. Do not invent method names. If you are unsure whether a method exists, write the code without that method.”

Or for null mishandling:

“Specify and enforce the null policy: every public method either rejects nulls with Objects.requireNonNull or documents in the Javadoc that nulls are valid input and what behavior they produce.”

Required deliverables

7. prevention-prompts.txt at the project root. One section per bug category you encountered. For each:
   • The category name.
   • The prompt template (3–8 sentences).
   • One example of “before” — the buggy code that would have been generated without this prompt clause.
   • One example of “after” — the cleaner code generated with this prompt clause. (You may run this experiment with your AI assistant and document.)

Aim for 4–6 prompt templates total.

Why this matters

Project 9 taught you to spec. Project 10 taught you to prompt precisely. Project 11 Medium puts those skills together: the best prevention for bug class X is a prompt clause that addresses X up front. The prompt template is the artifact that compounds — once you have it, you reuse it for the rest of your career.

Hard Tier (+up to 25% additional extra credit)

Pick one:

H1. Generate your own AI-buggy code

Pick a problem of comparable difficulty to the starters (a small class with 3–5 methods). Spec it. Prompt your AI for an implementation using a deliberately weak prompt so the output contains real bugs.

Then:

• Run your own full review and bug log on it.
• Write a test suite that catches every bug you can find.
• Hand the test suite to a classmate.
• Ask the classmate to write code that passes your tests but is still subtly wrong in some way.
• Iterate — strengthen your tests until your classmate gives up or you give up.

Document the cycle in hard-h1.txt. The final test suite + the buggy code + the iteration log are the deliverable. This is adversarial testing — a real-world skill.

H2. Build a “review-bot” prompt

Build and test a meta-prompt: a prompt you can send to your AI assistant along with a code snippet, asking the AI to play the role of code reviewer and produce a bug report.

Test your review-bot prompt on:

• The four starter files (does it find the same bugs you did?).
• Code you have written yourself in prior projects (does it find real bugs or spurious ones?).
• One or two clean, correct programs (does it correctly say “no bugs found” or does it hallucinate problems?).

Document the meta-prompt and the results in hard-h2.txt. Note where the AI was a useful second reviewer and where it was unreliable.

This rep produces a working tool. Many engineers use exactly this pattern in production. It is also a great reality check on AI’s limits as a reviewer of its own kind of work.

Submission

Submit one URL via the course portal.

Project structure:

project-11/
├── bug-log.txt                  (project-level summary + concatenated per-file logs)
├── prevention-prompts.txt       (Medium only)
├── hard-h1.txt or hard-h2.txt    (Hard only)
├── prompts.txt
├── bug-hunt-1/
│   ├── original.java
│   ├── fixed.java
│   ├── Tests.java
│   └── bug-log.txt
├── bug-hunt-2/ ...
├── bug-hunt-3/ ...
└── bug-hunt-4/ ...

Reflection block in bug-log.txt

<!--
Project 11 — Find the Bugs in AI's Code

Tier targeted:      Normal / Medium / Hard
Total bugs found:   N
Most common category: [category]
Hardest bug:        bug-hunt-N, [brief description] — found because [how]

Honest accounting:
  - I read each file with the ten-question checklist from §11.1.
  - I wrote my tests before looking at the bug answers (no answer key exists for me).
  - I made minimal fixes — no refactoring beyond what was needed to fix bugs.

What I'd change if I had another hour:
  [one sentence]
-->

What the grader will do

1. Open prompts.txt. Read it. (Project 11 should have less AI prompting than other Phase 2 projects — most of the work is reading. If your prompts log is long, the grader will look for what you were prompting for.)
2. Open the project-level bug-log.txt. Read the summary.
3. For each bug-hunt-N/ directory: a. Open original.java. Read. b. Open the answer key (grader-only). c. Open your fixed.java. Verify the bugs the key lists are fixed. d. Run your Tests.java. Verify your tests pass on fixed.java and fail on original.java. e. Open your per-file bug-log.txt. Verify classifications.
4. (Medium) Open prevention-prompts.txt. Verify the templates address the bug categories you encountered.
5. (Hard) Open hard-hN.txt. Read the iteration log.
6. Score.

Hints (Read These If You’re Stuck)

• “I found one bug fast and I’m not sure if there are more.” There are. Each file has at least two. Re-read with the ten-question checklist explicitly in front of you. The second bug is usually in a different category from the first.

• “My tests pass on original.java.” Then your tests don’t exercise the bug. The whole point is: your tests should fail on the original and pass on the fix. If they pass on both, you’ve tested the wrong thing.

• “I think there’s a bug but I can’t articulate why.” Run the code on a strange input. If it produces an output you can’t justify against the spec, that’s your articulation: “given input X, the spec says output Y, but the code produces Z.” That’s a bug report.

• “The AI told me this code is fine.” The AI is also sometimes wrong. Don’t outsource your review to the thing whose work you’re reviewing. (You may use the AI for secondary review — Hard H2 explicitly does this — but the primary review is yours.)

• “How minimal is ‘minimal’ for fixes?” As minimal as possible while still being a real fix. Adding a single null check is a minimal fix. Adding a 5-line guard clause is fine. Rewriting the method to use streams is not minimal — even if it would be cleaner. The principle: a colleague reviewing your diff in production should be able to say “yes, that fixes the bug” in 10 seconds.

• “How long should this take?” Normal: 4–6 hours. The reading is real. Medium: +2–4 hours for the prevention prompts and experiments. Hard: +4–6 hours for the adversarial cycle. Plan for it. This is not a one-evening project.

What Mastery Looks Like (Beyond the Rubric)

The rubric tells you what to do. Here’s what to aim for:

A great Project 11’s bug-log.txt is a document a security reviewer could trust. Each bug is named, located, classified, evidenced, and fixed. There is no hand-waving. The reviewer reading it does not need to re-find your bugs; you have shown your work.

A great Project 11’s tests encode the spec. Reading Tests.java for HistoryBuffer.java, a reader who has never seen the spec should be able to reconstruct it from the test names and assertions. The tests are not just “what catches the bug” — they are “what defines correctness.”

A great Project 11’s “what I almost missed” paragraph is the most useful piece of writing in your submission. It tells the grader (and your future self) about the failure mode you almost replicated. That meta-awareness is what senior engineers have and juniors don’t.

A great Project 11’s prevention-prompts.txt (Medium) is something you would actually paste into next month’s projects. If it isn’t, you wrote it for the grader, not for you.

When You’re Done

1. Re-read every bug-log.txt. Does each bug entry have evidence? Category? Fix? Test?
2. Run every Tests.java against both original.java and fixed.java. Confirm tests fail on the originals and pass on the fixes.
3. Confirm your prompts.txt honestly reflects how little (or how much) AI you used.
4. Submit.
5. Read Chapter 12. Now we talk about the senior’s most non-negotiable job.

Coach’s Note — Project 11 separates the students who can be senior engineers from the students who can’t, faster than any other project in this course. Reading carefully is not glamorous. It is also the single most-rewarded skill on every engineering team you will ever join. The hours you spend this week reading buggy code are an investment that compounds for the next 30 years. Treat them that way.

Sharpen your eyes. See you Monday.