Chapter 11 — Reps

This week the reps are not warmups around the milestone — they are the milestone, built one piece at a time. By Friday the test plan, the traceability matrix, the suite, the data, and the defect log all exist, because you built them here.

Ground rules

• Work on YOUR project. Every rep operates on your real requirements and your real code. PantryPilot is in the chapter so you have something to look at; it is not the thing you are testing.
• Derive tests from the requirement, not the code. Open docs/requirements.md first. If you have already read the implementation, have the assistant write the case from the requirement text alone.
• Log the defect before you fix it. Every one, including the thirty-second fixes. The log is graded, and it is the only evidence you found anything.
• Two columns, always. Severity and priority are separate, and each gets a one-line reason.
• Commit as you go. Each artifact lands in the repository with a dated commit message that names the requirement or defect ID.
• Reflections are written, in prose. A rep is not done until you have written the two to four sentences it asks for. Keep them in your hours log or a scratch document; the honest ones become the retrospective in Week 16.

Theme 1 — The plan and the promises

Rep 1 — Scope, out of scope, and six exit criteria

Copy code/test-plan-template.md into docs/test-plan.md. Fill sections 1, 2, and 6 today — before you look at how the build is doing.

Scope:        requirements ____ through ____, at commit ____________
Out of scope: 1. ______________________ because ______________________
              2. ______________________ because ______________________
              3. ______________________ because ______________________

Exit criteria (every line must be checkable by a stranger, without asking you a question):
  1. ______% of Must requirements have >= 1 passing acceptance test
  2. Zero open defects at severity ____ or above
  3. ______________________________________________________________
  4. ______________________________________________________________
  5. The full suite runs from `______________` and passes in CI
  6. ______________________________________________________________ (a number from your NFRs)

Reflect: Read criterion 6 back. Could a grader check it without you in the room? If not, it is an adjective wearing a number’s clothes — rewrite it.

Rep 2 — One acceptance criterion, one test case

Pick three acceptance criteria from your SRS — one you are confident about, one you are unsure about, one you have never exercised. Write a full test case for each in this shape.

TC-____  Verifies: ____-AC__   Level: unit / integration / acceptance   Data: ________
Preconditions: ______________________________________________
Steps:  1. ______________  2. ______________  3. ______________
Expected: ______________________________ (an observable value, not "it works")
Fails if: ______________________________ (the specific bug you are hunting)

Reflect: Which of the three was hardest to write, and why? A criterion you cannot turn into a test case is usually a criterion that was never testable — that is a Week-3 defect you just found for free.

Rep 3 — Partitions and boundaries

Take one requirement that contains a number, a date, a length, or a count. Produce the full partition table, then write a test at every boundary value.

| Partition          | Representative | Boundary values worth a test |
|--------------------|----------------|------------------------------|
| below the range    |                |                              |
| inside the range   |                |                              |
| above the range    |                |                              |
| absent / null      |                |                              |

Run the boundary tests. Expect at least one to fail. If none do, widen the partitions — you have probably only tested the middle of each class.

Reflect: How many boundary tests did this one requirement generate, and how many did you have before this rep? Multiply that ratio across your Must requirements. That is the size of the gap you are closing this week.

Theme 2 — Coverage that means something

Rep 4 — Build the traceability matrix and run the numbers

Copy code/traceability-matrix.csv into docs/traceability.csv and replace the sample rows with yours: one row per acceptance criterion, with its test case ID, level, status, and defect ID if it is failing. Then run the checker.

python3 coverage_report.py docs/traceability.csv

Use code/coverage_report.py. Record four numbers in docs/test-results.md: total requirements, verified requirements, verified-requirement coverage, and verified-Must coverage.

Reflect: Write your verified-requirement coverage down before you fix anything. Most students see something between 30% and 60% the first time. Whatever yours is, that number — not line coverage — is the honest state of your project on the day you ran it.

Rep 5 — Hunt the three bad buckets

The checker sorts your gaps into three lists. Work them in this order and write down what you find.

Reflect: How many requirements were in the “never run” bucket? That bucket is the one that quietly lies — before this rep, would you have counted those as covered?

Theme 3 — Data and execution

Rep 6 — Three families of test data

Build the happy, boundary, and hostile data sets described in 11.5. Generate them with a script and a fixed seed; commit both the seed and the output. Then make seeding a single command and put that command in the test plan’s Environment section.

seed --set happy    -> ____ records
seed --set boundary -> ____ records   (empty, one, max, zero, negative, earliest, latest)
seed --set hostile  -> ____ records   (unicode, emoji, apostrophes, 10k chars, injection
                                       strings, path traversal, wrong file type)

Run your existing suite against the hostile set.

Reflect: What broke? Almost everyone finds at least one crash or one silently truncated field here. Name the requirement it violates, and whether your specification ever said what should happen — because if it did not, you have found a requirements gap as well as a bug.

Rep 7 — Rewrite a bad defect report

Here is a real one, in the state most students leave them:

Search is broken on the items page, returns nothing sometimes even
though the item is definitely there. Happens on my laptop. Fix later.

Rewrite it with all seven fields from code/defect-report-template.md, inventing plausible specifics: environment, numbered reproduction steps, expected vs. actual, evidence, severity and priority with reasons, and traceability to a requirement and a test case.

Then do it again with a defect from your log — pick the sloppiest entry you have.

Reflect: Which field was hardest to supply for your own defect? If it was “steps to reproduce,” you have not actually isolated the bug yet, and you were about to start fixing it anyway.

Theme 4 — Judgment

Rep 8 — Triage ten defects with a real deadline

Take ten open defects (use your own; pad from the chapter’s examples if you have fewer). Assign severity and priority separately, each with a one-line reason. Then produce two lists.

FIX ORDER (in the hours you actually have left):
  1. ____  2. ____  3. ____  4. ____  5. ____

SHIPS WITH KNOWN DEFECTS (with the workaround a user needs):
  ____  workaround: ______________________
  ____  workaround: ______________________

Constraint: you have six hours. Not “as long as it takes.” Six. Make the list fit.

Reflect: Which defect did you want to fix that you cannot? Write one sentence defending the decision to a user who is affected by it. If you cannot write that sentence, re-triage — you got the priority wrong.

Rep 9 — Red, green, prove it

Pick one open defect. Do the loop in strict order, with three separate commits.

1. Write the failing test.       commit: "test: TC-___ reproduces DEF-___"
2. Fix the code.                 commit: "fix(FR-__): ______ — closes DEF-___"
3. Prove the test is real: run the new test against the PRE-FIX commit.
   It must be RED there. If it is green, the test never tested the bug.
4. Name the regression test in the defect log entry.

Reflect: Did step 3 pass on the first try — that is, was your test actually red against the old code? Report honestly. This is the step everyone skips, and it is the one that catches tests that verify nothing.

Theme 5 — The assistant, and the parts that will not sit still

Rep 10 — Grade the assistant’s test cases

Pick one requirement. Give an assistant the requirement text and acceptance criteria only — explicitly tell it not to look at your implementation — and ask for ten test cases with expected results.

Now audit all ten against your specification and score them:

| # | Asserts the right thing? | If not, what went wrong?                       |
|---|--------------------------|------------------------------------------------|
| 1 |  yes / no                | change detector / empty assertion / invented    |
|   |                          | API / wrong oracle                              |

Count the wrong ones. Then run the same prompt again with your code included in the context, and count again.

Reflect: Report both counts. Was the second run worse? It usually is — that is the change detector, and it is why the requirement, not the code, is the right input. Record the tool and this use in docs/ai-usage.md.

Rep 11 — Test the part that will not sit still

If your project has a non-deterministic component — a model call, a recommendation, anything sampled — write these four now. If it does not, write the failure-path tests for your riskiest third-party dependency instead; the shape is identical.

1. STRUCTURE:  output parses; required keys present; types and ranges correct
2. PROPERTY:   an invariant true of EVERY valid output
               e.g. "no suggestion names an ingredient not in the input list"
3. GOLDEN SET: ____ fixed inputs, threshold committed in the test plan:
               "at least ____ of ____ produce a valid result"
4. FAILURE:    429 / timeout / malformed response / missing key / provider down
               -> each has a required behavior, tested against a STUB

Stub the provider by default. Put any real-provider test behind an explicit flag so CI never spends your quota.

Reflect: Which of the five failure conditions did your code not handle at all before this rep? That one was going to happen live in Week 16, on someone else’s network.

Rep 12 — The pesticide check

Your suite is green. Spend forty-five minutes in an unscripted exploratory session: use the software the way a hostile, impatient, confused user would. Double-click everything. Submit empty forms. Go back. Refresh mid-operation. Paste 10,000 characters. Log every defect you find.

Reflect: How many defects did a green suite fail to catch? Beizer’s pesticide paradox says the bugs your existing methods can find are already gone. Name one kind of test you are not currently doing, and add it to the test plan’s Levels section.

Done? One Last Thing.

The verification dress rehearsal. Run the week end to end, exactly as your Week-16 grader will:

1. git clone your own repository into a fresh directory. Not a pull — a clone, somewhere else on disk.
2. Follow only what README.md says, and seed the data with the one documented command.
3. Run the full suite with the one documented command. Record the result in docs/test-results.md: pass count, fail count, duration, commit SHA, date.
4. Regenerate the coverage numbers with code/coverage_report.py and paste them into docs/test-results.md beside the run.
5. Open docs/test-plan.md and check yourself against your own exit criteria, line by line. Tick the ones you meet. Do not edit the criteria to match the result.
6. For every criterion you missed, do one of two honest things: fix it, or write the “ships with known defects” entry that says what you are shipping and why.
7. Commit everything with milestone 11: test plan, suite, results, defect log.

If that clone runs clean and your exit criteria are either met or honestly annotated, you have done real verification — the thing most capstones only claim.

Up next: Milestone 11