Project 8 — Concurrent Data Pipeline (MIDTERM)

“But all things should be done decently and in order.” — 1 Corinthians 14:40

Chapter: 8 — Concurrency, Threads, and Midterm Review Due: End of Week 8 — submitted at the end of the live 60-minute session Submit: A link to your code — a public GitHub repo URL with your pipeline source files and README.txt. See Appendix A for the local toolchain + git workflow. Allowed tools: Python 3.10+, the standard library (threading, queue, multiprocessing, time), a non-AI editor, and this textbook (paper or non-interactive PDF — no clickable links, no chat assistant). AI — Phase 1 (wk 1–8): AI is OFF. No ChatGPT, Claude, Copilot, Cursor, any agent, any assistant. This is the cumulative exam of the cost-and-structures half of the course. The whole point is that the skill is in your hands.

EXAM CONSTRAINTS — read before the clock starts.

• 60 minutes. One sitting. The clock does not stop. (Some institutions allow 75; your instructor will announce the exact time.)
• Closed-book except this textbook. No internet. No AI of any kind. No friend’s code, in person or by message. No notes that aren’t your own course work.
• Open textbook. You may use this book — especially §8.6 (queue.Queue), §8.8 (the cost table), and code/producer_consumer.py as a reference for the shape, not to copy line-for-line.
• Normal tier is passing. Medium and Hard are extra credit on top.

The Setup

A small ministry runs a nightly job: it has a pile of work items — say, prayer-request records to scrub and index, or memory verses to validate — and it needs them processed before morning. Right now the job is a single loop that handles one item at a time, and as the pile has grown, “before morning” has started slipping past sunrise.

You are the engineer they called. The fix is a pipeline: producers that read the work items onto a shared, thread-safe queue, and a team of consumers that pull items off and process them — many hands, working at once, without losing a single record or processing one twice.

But you also have to do the thing an architect does that a coder doesn’t: you have to know what the concurrency actually buys. The ministry’s work has two flavors — some items mostly wait (an external lookup, a slow disk read) and some items mostly compute (a heavy local transformation). One of those flavors gets faster with threads. One does not, no matter how many threads you add. If you reach for the wrong tool, you will write more code and ship a job that is exactly as slow as before. The exam is whether you can build the pipeline correctly and reason correctly about its cost.

Sixty minutes. Many hands, one coherent result, no chaos. Begin.

Learning Targets

By completing this project, you will demonstrate that you can:

• Build a correct multi-threaded producer/consumer pipeline using queue.Queue.
• Guarantee exactly-once processing — no lost items, no duplicates — and prove it with an assertion.
• Use a threading.Lock correctly to guard shared mutable state.
• Shut a pipeline down cleanly with the poison-pill pattern.
• Measure and explain the GIL: that threads speed up I/O-bound but not CPU-bound work.
• Use multiprocessing to achieve real parallelism for CPU-bound work.
• Make and defend the threads-vs-processes-vs-event-loop right-tool decision in writing.

Normal Tier

Goal: In 60 minutes, build a correct producer/consumer pipeline. One or more producer threads put “work items” on a thread-safe queue; one or more consumer threads take them off and process them. No race conditions. No lost items. Prove that every item produced is consumed exactly once.

Required features

1. A thread-safe queue. Use queue.Queue to hand items from producers to consumers. (You may instead use a shared structure guarded by a threading.Lock if you want to prove you can — but queue.Queue is the recommended, correct tool, and choosing it is the right-tool point.)
2. Multiple producers. At least 2 producer threads, each putting a known number of items on the queue (e.g., 1,000 each → 2,000 total).
3. Multiple consumers. At least 2 consumer threads, each pulling items off the queue and “processing” them (any deterministic work — a counter increment, a small computation — is fine).
4. No races. Any shared mutable state you keep on the side (e.g., a count of items consumed) must be guarded by a Lock, or recorded in a thread-safe way. The hand-off itself is safe because the queue handles it.
5. Clean shutdown. Use the poison-pill pattern: after all real work is enqueued, put exactly one sentinel (e.g., None) per consumer so each consumer exits. join every thread.
6. A correctness proof. At the end, assert that the number of items consumed equals the number produced (no losses, no duplicates) and that the queue is empty. The assertion is part of the deliverable — it is how the grader sees that you understood “exactly once.”
7. It runs. python3 pipeline.py runs to completion, prints the produced and consumed counts, and the assertion passes.

A pipeline_starter.py skeleton is in this chapter’s code/ folder — it has the thread wiring in place and TODOs where the logic goes. You may start from it.

Example output

Produced: 2000
Consumed: 2000
Every item produced was consumed exactly once. No races.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

M1. Threads speed up I/O-bound work — proven

Make the consumer’s processing I/O-bound by simulating a wait (time.sleep(0.01) per item). Run the pipeline two ways and time each with time.perf_counter(): once with 1 consumer thread, once with several (e.g., 8). Show in numbers that more threads finish the same workload substantially faster. Print the times and the speedup.

M2. Threads do NOT speed up CPU-bound work — proven, with the GIL explained

Now make the consumer’s processing CPU-bound (a tight pure-Python loop per item, no sleep). Run again with 1 thread vs several, time both, and show that more threads give little or no speedup. In your README.txt, explain in 2–4 sentences why, using the words “GIL” and “released during I/O.” This is the central right-tool lesson of the week, and the explanation is graded as much as the numbers.

Hard Tier (+up to 25% additional extra credit)

H1. Solve the CPU-bound case with multiprocessing

Take the CPU-bound workload from M2 and parallelize it for real using multiprocessing (a Pool, or processes communicating via a multiprocessing.Queue). Time it against the serial and threaded versions. Show a genuine speedup that threads could not deliver. Remember the if __name__ == "__main__": guard. (A clear, correct written design for the multiprocessing version earns partial H1 credit if you run out of clock — but a running implementation earns full credit.)

H2. The decision memo — the judgment an agent can’t make for you

Write MEMO.docx: a one-page architect’s decision memo with a table mapping workload to right tool, backed by your own measured numbers from M1, M2, and H1:

The third row is your bridge into Week 9: explain, in 2–3 sentences, why a single-threaded event loop is the right tool for thousands of mostly-idle connections, and the one workload it is wrong for (CPU-bound work that freezes the loop). Cite your own numbers for the first two rows. The memo is the heart of the Hard tier — it is the architect’s deliverable that no agent can produce for you, because it requires your measurements and your judgment about this workload.

Submission

Submit one URL via the course portal: a public GitHub repo containing:

1. pipeline.py — your Normal-tier pipeline (and the M1/M2/H1 timing variants, in this file or clearly-named companions like bench_io.py, bench_cpu.py, cpu_multiprocessing.py).
2. README.txt — your reflection (template below).
3. MEMO.docx — for Hard tier (H2).
4. The program left runnable — python3 pipeline.py runs and the assertion passes.

# Project 8 — Concurrent Data Pipeline (Midterm)

**Tier targeted:**  Normal / Medium / Hard
**Features done:**  (list)
**Correctness proof:**  (where in the code, and what it asserts)
**Race protection:**  (queue.Queue and/or which Lock guards what)
**Shutdown:**  poison pill — yes / no
**GIL finding (Medium):**  threads gave __x on I/O, __x on CPU
**Multiprocessing finding (Hard):**  __x speedup on CPU-bound
**What I learned:**  (one paragraph)
**What I'd change:**  (one sentence)
**AI usage:**  NONE — Phase 1, midterm, closed-AI.  Signed: <your name>

Add the same fields, condensed, as a comment block at the top of pipeline.py.

Hints (Read Before You Begin)

• Reach for queue.Queue first. It does the locking for you. Hand-rolling a list + lock is more code, more bugs, and earns the same points. The right tool is the queue — choosing it is the lesson.
• The exactly-once proof is the soul of Normal tier. Don’t just run the pipeline — count what was consumed (under a lock) and assert it equals what was produced, and that the queue is empty. A pipeline that “runs” but can’t prove correctness is half the points.
• Watch for the race inside your race fix. The queue makes the hand-off safe. It does not make your side counter safe. Two consumers doing consumed += 1 with no lock will lose updates — the exact bug from §8.4, hiding inside your solution. Guard that counter.
• Send one poison pill per consumer, after all real work. Too few and a consumer hangs on an empty queue; forget them and the program never exits. Then join everything.
• For Hard tier, guard the entry point. All multiprocessing spawning goes inside if __name__ == "__main__":, or children re-import and spawn recursively.
• Budget the hour. ~25 min for a correct, proven Normal tier (lock it in first). ~15 min for M1+M2 timings. ~20 min for H1+the memo. Don’t chase Hard with Normal unfinished — a proven Normal beats a broken Hard.

What Mastery Looks Like (Beyond the Rubric)

A great Project 8 is short and certain. The pipeline is maybe forty lines. The producers produce, the consumers consume, the poison pills shut it down, and the final assertion proves — not hopes — that every item made it through exactly once. There is no clever trick; there is the right tool (queue.Queue) used the right way, and a proof that it worked.

A great Project 8 reasons correctly about cost. When the grader reads your Medium-tier numbers, they see a 4x-ish speedup on I/O and a flat line on CPU, and your two sentences explain why with the GIL — not vaguely, but precisely: the lock is released while a thread waits, held while it computes. The architect who wrote those sentences will never again throw threads at a CPU-bound loop and wonder why it didn’t get faster.

And a great Hard tier ends in a memo that an agent could not have written, because it is built on your measurements of this workload and your judgment about which tool the constraint demands. That memo is the whole course in miniature: the right tool is a decision made before code is written, driven by the constraints of the problem, and defended with numbers.

Coach’s Note — This is the first test of Coding 3 where the agent is off and you are alone with the keyboard. You will feel the absence in the first ten minutes. Hold the discomfort — it is the test working. If you typed the reps and built the structures with your own hands, your hands will start moving by minute twenty, and the pipeline will take shape under them. If they don’t, the midterm is telling you the truth, and you have eight weeks before the final to make it false. Either way, the score is a diagnostic, not a verdict. Use it like one.

When You’re Done

1. Run python3 pipeline.py. Confirm the produced and consumed counts match and the assertion passes.
2. Run it five more times. A correct pipeline passes every time — the variation that betrays a race must not appear.
3. (Medium) Run your I/O and CPU benchmarks. Confirm the I/O one speeds up with threads and the CPU one doesn’t, and that your README explains it.
4. (Hard) Run the multiprocessing version. Confirm the speedup, and that your MEMO.docx cites your own numbers.
5. Push to GitHub. Submit the URL before the clock runs out.
6. After the exam, breathe — and read Chapter 9. Phase 2 begins. Your first server, the agent comes on, and the event loop you reasoned about in the memo becomes something you build.

A theological footnote. Paul wrote 1 Corinthians 14 to a congregation drowning in its own gifts — everyone speaking at once, real worship turning to noise. His instruction was not fewer voices but ordered ones: “all things should be done decently and in order,” because “God is not a God of confusion but of peace.” That is, with startling precision, the engineering of this midterm. Many threads, real work, and a discipline of coordination — a queue, a lock, a clean hand-off — that turns simultaneous effort into one coherent result instead of a corrupted count. Order is not the enemy of many hands at work. Order is the only thing that lets many hands work as one. When your pipeline runs clean — every item through exactly once, no chaos — you have built a small picture of how a body is meant to work.

See you in the exam room.