Milestone 11 — Test Plan, Test Suite & Defect Log

“Who can discern his errors? Declare me innocent from hidden faults.” — Psalm 19:12 (ESV)

Chapter: Chapter 11 — Testing, Quality, and the Defect Log Week: 11 of 16 — the verification phase Due: End of Week 11, committed and pushed Submit: Everything in your project repository, committed with the message milestone 11: test plan, suite, results, defect log Where it lands in the final package: docs/test-plan.md, docs/traceability.csv, docs/test-results.md, docs/defect-log.md, and the tests/ directory. This milestone feeds the test suite and results line and the defect log line of the Week-16 rubric — the 50% component. See Appendix C for the exact point values. AI policy: Open, with the spine rule. An assistant may propose test cases, edge cases, and partitions — this is one of its strongest uses and you should use it. Only you may approve an assertion, because an assertion is a claim about what your software is supposed to do and you are the one who signed the requirements. Record the tool, what you used it for, and what you rejected in docs/ai-usage.md.

The Setup

Your core works. You demoed it last week and it did what you said it would do.

Now imagine somebody else in the room. Not a friendly grader — a QA engineer on their third coffee, who has never seen your project, who has your requirements specification open on one monitor and your running application on the other, and whose entire job this afternoon is to find the places where those two things disagree. They will not use your software the way you use it. They will type an apostrophe into a name field. They will set an expiry date to today, then to yesterday, then leave it blank. They will submit the form twice. They will unplug the network mid-request.

That person is you, this week, and the hardest part is that you already know where the bodies are buried. You know which error path returns a 200 with an empty body. You know which date comparison you never checked at the boundary. You have been carrying those around since Week 9 as a low hum of unease.

This milestone converts that unease into artifacts: a plan that says what “done” means in numbers, a matrix that maps every promise to the test that verifies it, a suite that runs from one command, and a log that records — in your own name, on the public record — every fault you found. Four documents. None of them can be reconstructed in Week 15 from memory, and all of them are worth points that do not come back.

Like every milestone in this course, Milestone 11 is graded twice — and by now you know exactly what that means. It carries its own points in the 25% milestone bucket, and these artifacts then go straight into the Week-16 package that carries 50% of your grade. Producing them this week costs you fifteen hours. Producing them in Week 16, alongside deployment, documentation, the handoff package, and a thirty-minute talk, costs you the grade.

Learning Targets

You will demonstrate that you can:

• Write a test plan with a scope, an honest out-of-scope section, named levels, a reproducible environment, and exit criteria a stranger could check.
• Trace every functional and non-functional requirement to the test cases that verify it, and report verified-requirement coverage rather than line coverage.
• Derive test cases from requirements, not from code — including equivalence partitions, boundary values, and negative cases.
• Build test data that resembles reality: happy, boundary, and hostile, generated reproducibly, with no real personal data.
• Write a defect report a stranger could act on, with all seven required fields.
• Triage with severity and priority as separate axes, and produce a defensible fix order against a real deadline.
• Practice regression discipline: failing test first, then the fix, then proof the test was ever red.
• Decide when quality is good enough — and document what ships with known defects.

The weekly status block — paste this into your submission comment

Every milestone submission carries these five lines. It takes two minutes, it is the same five lines every week, and it is the difference between an instructor who can help you in Week 6 and one who finds out in Week 14. Copy it into the Canvas submission comment when you submit — not into a document.

Shipped:     what actually works this week, in one line
Cut:         what I dropped or deferred, and where it went (backlog / change request / gone)
Hours:       planned N / actual N        (from docs/hours-log.csv)
Blocked on:  the one thing in my way, or "nothing"
Ship confidence: green | amber | red — if amber or red, the one thing that would move it

Say red when it is red. A red in Week 6 is a conversation and a scope cut. A red discovered in Week 15 is a grade. Nobody has ever been penalised in this course for an honest amber; the only expensive answer is the green that was not true.

Normal Tier

Required deliverables

1. docs/test-plan.md — from code/test-plan-template.md. All nine sections filled: scope (named commit), out of scope with reasons, levels, environment, entry criteria, exit criteria as checkable numbers, risks to the testing itself, schedule, and your signature.

2. docs/traceability.csv — from code/traceability-matrix.csv. One row per acceptance criterion: requirement ID, MoSCoW priority, criterion ID, test case ID, level, status, defect ID. Every Must requirement has at least one test case. Requirements with none appear in the matrix as gaps rather than being quietly omitted.

3. An automated test suite in tests/, run by one documented command, passing in the CI pipeline you built in Week 9. At minimum:

   • unit tests covering your core logic, with boundary and negative cases for every requirement that has a number, a date, a length, or a count in it;
   • at least three integration tests crossing real seams from your Week-6 design — database, third-party API, model provider, file system;
   • acceptance tests derived one-to-one from acceptance criteria, at least one per Must requirement, each naming the criterion it verifies.

4. A test data set, committed, in three families — happy, boundary, hostile — generated from a fixed seed by a script, seeded with one command, containing no real personal data.

5. docs/test-results.md — the record of your last full run: commit SHA, date, command, pass/fail counts, duration, and the four coverage numbers from code/coverage_report.py (total requirements, verified, verified-requirement coverage, verified-Must coverage).

6. docs/defect-log.md — from code/defect-report-template.md. At least eight defects, each with all seven fields, severity and priority assigned separately with a stated reason, and — for every defect marked fixed — the named regression test that reproduces it.

Normal-tier rubric (out of 100)

If Your Project Does Not Look Like a Web Application

This course is stack-agnostic and so is this milestone. The three levels are required; what they contain depends on the shape of what you built. Find your row, and write the mapping into your test plan’s Levels section so a grader reads the same shape you built.

Two things do not change, whatever your row says. Every Must requirement gets an acceptance test, and if your acceptance level is manual, the “test case” is a written script with numbered steps and expected results that someone else could execute — not a memory of you clicking around. Every defect gets logged, whether it was found by a machine or by your own hands.

Medium Tier (+up to 25% extra credit)

Depth in verification. Pick what actually fits your project — two done well beat five done thinly.

• A coverage gate in CI. Add a line-coverage tool to the pipeline with a threshold you choose and defend in the test plan. Report it underneath your verified-requirement coverage, not instead of it.
• Mutation testing on your core module. Tools exist for most ecosystems (Stryker for JavaScript and .NET, PIT for the JVM, mutmut and cosmic-ray for Python, and others). It deliberately introduces small faults and reports which ones your suite failed to catch. It is the sharpest available answer to “are my assertions real?” — and it is humbling.
• Property-based tests. Three or more invariants that must hold across generated inputs (Hypothesis, fast-check, jqwik, or your ecosystem’s QuickCheck descendant), each stated in prose in the test plan before it is stated in code.
• A performance test against your own NFR. Take a numeric target from your Week-4 non-functional requirements and write a repeatable measurement of it, with the data set and machine named. Record the number even if it fails the target — especially if it fails.
• An accessibility pass as tests. If your project has a user interface, run an automated accessibility checker over your main flows and log every finding as a defect with a severity. Automated tools catch only a fraction of real barriers, so add one manual check — keyboard-only navigation through your primary task — and log what you find. See w3.org/WAI for what the criteria mean.
• A contract test for a third-party dependency. Capture a real response, commit it as a fixture, and test that your parser survives that shape plus a 429, a timeout, and malformed content.

Hard Tier (+up to 25% additional extra credit)

A two-to-three page Quality Judgment Memo, in docs/quality-memo.md, addressed to the engineer who will inherit this project.

An assistant can generate test cases all day. It cannot do this, because this requires owning the consequences of a decision made with incomplete information and a deadline. Four sections, each answering a question you cannot look up:

1. Defend your exit criteria. Why these numbers? Why is zero-open-S2 the right bar for your project and not zero-open-S3? Name the user impact behind each threshold. Then state plainly whether you met them, and where you did not.

2. Defend what ships with known defects. For every open defect in v1.0, state the severity, the priority, the workaround, and — this is the part — the user who is affected and why you decided their inconvenience is acceptable this semester. Write it as though they will read it, because your handoff guide in Week 14 means someone will.

3. Name what you chose not to test, and what that costs. Every test plan has an out-of-scope section. Yours is a set of bets. Name the two you are least comfortable with, describe the failure that would prove you wrong, and say what a successor should test first when they pick this up.

4. Report your assistant’s review burden honestly. From Rep 10: out of ten generated test cases, how many asserted the wrong thing, and what kind of wrong were they? Then draw the conclusion — at that error rate, what is the fastest you are actually allowed to accept generated tests without reading every assertion? Cite docs/ai-usage.md.

Graded on judgment, specificity, and candor. A memo that says everything went well earns nothing; a memo that names a real weakness and defends a real tradeoff earns full marks even when the tradeoff is uncomfortable.

Submission

Everything lives in your repository. The structure a grader expects:

your-project/
├── README.md
├── docs/
│   ├── requirements.md          # from Weeks 3-4
│   ├── test-plan.md             # NEW this week
│   ├── traceability.csv         # NEW this week (grown from the Week-4 matrix)
│   ├── test-results.md          # NEW this week
│   ├── defect-log.md            # NEW this week
│   ├── quality-memo.md          # Hard tier only
│   ├── ai-usage.md              # updated with this week's uses
│   └── hours-log.csv            # updated
├── tests/
│   ├── unit/
│   ├── integration/
│   ├── acceptance/
│   └── data/                    # happy / boundary / hostile + the seed script
└── .github/workflows/           # or your CI equivalent — the suite runs here

Commit message: milestone 11: test plan, suite, results, defect log. Push before the deadline; the timestamp on the push is the timestamp on the milestone.

Suggested Pacing

Fifteen hours, and the order matters more than the total. Do not let the first acceptance pass get squeezed to the end — it is where most of your defects will come from.

If Thursday finds nothing, Thursday went wrong — not the software. Go back to the hostile data set and the boundaries.

Hints

• Write the exit criteria on Monday. Criteria written on Friday are always, mysteriously, criteria the current build happens to meet.
• Start with the requirement that scares you. Not the one that is easiest to test — the one you have been avoiding since Week 9. That is where the defects are, and finding them on Monday is much cheaper than finding them on Thursday.
• Expect your first coverage number to be low. Somewhere between 30% and 60% verified-requirement coverage is normal on the first run. That number is not a judgment of you; it is the measurement you have never taken before.
• Log before you fix. Always. Even the thirty-second fixes. A defect log of three entries reads as no testing at all.
• Do not fix everything. You have fifteen hours and an S4 typo is not worth an S2’s slot. The triage is the skill.
• Pin the clock, seed the randomness, stub the network. Three habits that prevent the entire category of tests that pass today and fail in the presentation room.
• Keep the assistant away from your implementation when generating tests. Give it the requirement text only. Otherwise it will write tests that lovingly preserve your bugs.
• Templates and worked good/bad examples for all four documents are in Appendix B; the vocabulary is in Appendix E.

What Mastery Looks Like

A master’s verification package has a particular smell to it, and a reviewer picks it up in about ninety seconds.

The exit criteria are numbers, and at least one of them was not met — and that fact is written down beside the number rather than erased. The traceability matrix contains gaps, clearly marked, because the author would rather show you an unverified requirement than pretend it does not exist. The defect log has fourteen entries, not four, and three of them are marked “will not fix” with a reason a user could read without being insulted. Severity and priority disagree on at least two lines, which is how you know the author understood the difference. Every fixed defect names a regression test, and those tests have real names like test_expiry_window_includes_today rather than test_bug_fix_2.

And somewhere in docs/test-results.md there is a sentence like: “Verified-requirement coverage is 84%. FR-16 and NFR-07 remain unverified; both are documented in the known-issues list with the tests a successor should write first.”

That sentence is worth more than a green badge, because a green badge is a claim and that sentence is a confession with a plan attached. Anyone who has shipped software knows which one to trust.

Coach’s Note — You will be tempted, somewhere around Thursday, to soften an exit criterion so the build passes it. It will feel like a small edit. It is not. That edit is the exact moment a professional becomes a person performing professionalism, and it is the habit that follows you into a job where the number protects a real user. Miss the criterion. Write down that you missed it. Ship the honest list. That is the whole lesson of the week, and it costs you three rubric points and buys you a career.

When You’re Done

• docs/test-plan.md complete: scope, out-of-scope with reasons, levels, environment, entry criteria, numeric exit criteria, risks, schedule, signature
• docs/traceability.csv covers every requirement; every Must has at least one test case; gaps are visible
• Suite runs from one documented command and is green in CI on a clean checkout
• Unit tests include boundary and negative cases for every numeric, date, or length requirement
• At least three integration tests cross real seams
• One acceptance test per Must requirement, each naming the acceptance criterion it verifies
• Test data: happy, boundary, hostile — reproducible from a committed seed, no real personal data
• docs/test-results.md records the run and the four coverage numbers
• docs/defect-log.md has eight or more defects with all seven fields
• Severity and priority assigned separately, each with a reason
• Every fixed defect names its regression test, and that test was verified red against the pre-fix commit
• Exit criteria checked against the real build; misses annotated, not edited away
• docs/ai-usage.md and docs/hours-log.csv updated
• Weekly quiz taken
• Everything committed and pushed

A theological footnote. “Who can discern his errors? Declare me innocent from hidden faults” (Psalm 19:12, ESV). The psalmist’s problem is not that he refuses to face his faults. It is that the worst of them are hidden — invisible from the inside, precisely because they are his. That is the exact condition of a developer testing his own code, and it is why sincerity is not a method. You cannot find, by trying harder, the fault you cannot see. You need something outside yourself: a written requirement you cannot renegotiate at midnight, a matrix that does not care how the week went, a machine that runs your claims and reports without flattery. That is what a test suite is — a small instrument of honest examination, built by you, when you were being truthful, to check on you later, when you will be tired and motivated. And notice the psalm’s ending in Chapter 11’s epigraph: search me… and lead me in the way everlasting (Psalm 139:23–24, ESV). The searching is for the sake of the leading. Your defect log is not a monument to failure. It is the map of what to fix — and the reason a stranger will be able to trust the thing you hand them in Week 16.