Chapter 8 — Reps

Conditioning, not grading. This week’s reps are the measurement discipline itself: building and reading a harness, breaking it on purpose, diagnosing counter sets cold, ranking the same four parts four different ways, and auditing your own log the way a reviewer would.

Ground rules:

1. Type every command yourself. No copy-paste from the chapter. Your fingers have to learn --reps, --warmup, perf stat -e, and the shape of a median-plus-IQR line, because in the capstone you will be doing this against a deadline.
2. Run everything. Every rep below runs on Workbench L (a laptop with g++ and Python 3) and on Workbench B (the browser workbench, no install, no admin). Nothing here requires a phone. Where a device would let you go further, it is marked optional.
3. Predict before you measure. Every measuring rep: write down the number you expect — or at least the ordering you expect — before you run anything. Then run it. The gap between your prediction and the result is the entire lesson; a rep where you were right teaches you almost nothing, and a rep where you were wrong is worth an hour of reading.
4. AI policy — explain, never source. You may ask a model to explain a flag, a statistic, or a mechanism. You may not ask it for a hardware figure: no cache sizes, no clock speeds, no bus widths, no TOPS ratings, no benchmark scores. Any number about real silicon in your write-up carries a primary-source citation — a vendor specification page, an architecture reference manual, or your own measurement. End every AI-touching rep with the one-line AI-usage note Appendix D defines.
5. Log as you go. Open a measurements.xlsx from Appendix C’s template before Rep 1, not after Rep 11. Rep 11 audits it.

Everything you need is in this chapter’s code/ folder. Build the harness once and leave it built:

g++ -O2 -std=c++17 -o bench_harness code/bench_harness.cpp

Reps 1–3: The Harness and the Discipline

Rep 1 — Build the harness and read the dispersion

Run the four built-in kernels with a real repetition count:

./bench_harness --reps 31 --warmup 3 > bench.csv
cat bench.csv

Predict first, in writing, before you look. Rank the four kernels — alu, seq, stride, chase — from fastest to slowest per operation, and say why for each. Commit to the ordering. (Read code/bench_harness.cpp if you need to know what each kernel does; do not run it first.)

Now run it and compare. Almost everyone gets alu wrong. Write three sentences: which kernel you misplaced, what you had assumed about it, and what the real explanation is. Then read the iqr_ns column and answer the question that actually matters — for which kernels is the IQR small enough that you would be willing to defend a 10% difference between two builds? Record the run in measurements.xlsx with your machine, power state, and thermal condition.

Rep 2 — Delete the optimizer barrier and watch the benchmark lie

Copy the harness to a scratch file and sabotage exactly one thing — the barrier:

cp code/bench_harness.cpp nobar.cpp
# In nobar.cpp, change the body of do_not_optimize to discard its argument:
#     static inline void do_not_optimize(std::uint64_t v) { (void)v; }
g++ -O2 -std=c++17 -o nobar nobar.cpp
./nobar --reps 11

Predict first: which kernels do you expect to change, and by how much? Most people predict “the ALU one will get faster.”

Now look at what you actually got. Then rebuild the unmodified harness at -O0 and compare that too:

g++ -O0 -std=c++17 -o bench_O0 code/bench_harness.cpp && ./bench_O0 --reps 11

Write four sentences: what the numbers from nobar claim, why they are impossible, exactly which compiler transformation produced them, and — the professional question — what would you have concluded if you had only ever run nobar and never seen the barriered version? Then say why -O0 is also the wrong answer for a benchmark you intend to report.

Rep 3 — The cold-run lie: first run versus steady state

This rep needs no special hardware; a laptop throttles.

# Cold: let the machine idle for 5 minutes first, then a single short run.
./bench_harness --reps 5 --warmup 0 --kernel chase

# Immediately: eight back-to-back long runs, no cooldown.
for i in 1 2 3 4 5 6 7 8; do ./bench_harness --reps 31 --kernel chase; done

Predict first: by what percentage will median_ns drift from the first long run to the eighth? Write a number, not a direction.

Run it, then plot the eight medians by hand on paper (eight points, one axis). Write down: the drift you observed, whether it looks like thermal drift or noise, and how you would tell the difference with one more experiment. Then answer the honesty question in one sentence: if you had reported only the cold --warmup 0 run, what claim would you have made, and would it have been false? (Note in measurements.xlsx whether you were on battery or mains — it matters, and the capstone requires it.) Optional (Workbench D): repeat on a phone using the soak protocol in Appendix C and compare the shape of the curve.

Reps 4–6: Counters and Diagnosis

Rep 4 — Diagnose three counter sets cold

Open code/sample-perf-stat.txt in an editor. It contains three runs. Do not run the tool yet.

For each of the three, by hand and on paper: compute IPC, compute L1D MPKI and last-level MPKI, compute the branch miss rate, and write a one-line diagnosis naming the bottleneck. You need a calculator, nothing more. (MPKI = misses ÷ (instructions ÷ 1000).)

Now check yourself:

python3 code/counters.py code/sample-perf-stat.txt

Write down which of your three diagnoses matched, which did not, and — for any miss — the specific number you under-weighted. Then the harder question, worth more than the arithmetic: the tool’s diagnosis is a documented heuristic with thresholds. Find the thresholds in code/counters.py, and name one realistic workload for which each of them would give the wrong answer. A diagnostic rule you cannot break is a rule you do not understand.

Rep 5 — Utilization versus IPC: the busy core that is not working

Run the two kernels that sit at opposite ends of §8.3’s table, and watch your system monitor while they run:

./bench_harness --reps 51 --kernel alu    # dependent ALU chain
./bench_harness --reps 51 --kernel chase  # dependent pointer chase

Predict first: what CPU utilization will your system monitor report for each? Then run them and look.

You will see essentially the same utilization for both — near 100% of one core — while the per-operation costs differ by roughly an order of magnitude. Write three sentences: what utilization actually measured in each case, what it failed to distinguish, and which two additional numbers you would need to tell the two situations apart from telemetry alone.

Then the transfer question. You are handed a production dashboard showing a service at 95% CPU. List, in order, the three questions you ask before anyone is allowed to say the words “we need faster cores.”

Rep 6 — Compute MPKI by hand, then check yourself

If you are on Linux with counter access, produce your own counter set instead of using the sample:

perf stat -e cycles,instructions,branches,branch-misses,\
L1-dcache-loads,L1-dcache-load-misses,LLC-loads,LLC-load-misses \
  ./bench_harness --reps 21 --kernel chase 2> chase.txt
python3 code/counters.py chase.txt

On macOS use Instruments / xctrace; on Workbench B, or if counters are unavailable, re-use the ./matmul_ijk block from the sample file. Either way:

1. Compute IPC and LLC MPKI by hand first.
2. Check against the tool.
3. Then look at what perf printed next to each event. If any event shows a multiplexing percentage below 100, say which ones, and explain in one sentence why a miss rate built from two of those events could be wrong even though each count is individually close.

Finally: re-run with only three events (cycles,instructions,LLC-load-misses) and compare. Did anything move? Write one sentence on what that tells you about the first run.

Reps 7–9: Comparing Honestly

Rep 7 — Rank four SoCs four different ways

python3 code/compare.py

Predict first, from the raw table only (cat code/soc-compare.csv): before running the tool, write down which of SoC-A..SoC-D you would recommend for (a) a flagship phone, (b) a fanless tablet, (c) a wearable, and (d) a device that runs a sustained on-device model all day. Four recommendations, four one-line reasons.

Now run the tool and read all five rankings. Write down: how many of your four recommendations survived, which metric you had been implicitly optimizing for without saying so, and the single sentence you would put at the top of a comparison chart so that a reader could not be misled by it.

Then the integrity check, and answer it in writing: SoC-A..SoC-D are fictional and the file says so on its first line. Name two conclusions you are therefore not entitled to draw from this exercise, however satisfying they would be.

Rep 8 — Derive the memory bandwidth yourself

No tool for this one. Paper and a calculator.

Using bytes/s = data rate (MT/s) × bus width (bytes), compute the peak theoretical DRAM bandwidth for each row of code/soc-compare.csv. Check your four numbers against the GB/s column that compare.py prints.

Then answer three questions in writing:

1. Two of the four parts share a data rate. Why does one of them have twice the bandwidth? State the general rule in one sentence.
2. This is a peak theoretical figure. Name three separate reasons a real workload will not achieve it.
3. Now the citation drill. Pick one real, currently-shipping SoC and find its memory interface width and data rate from a primary source — the vendor’s own specification page or documentation. Record the figures, the exact URL, and the date you retrieved it, in the form the capstone requires. If you cannot find a primary source for one of the two figures, write “not published by the vendor” — that is a correct and valuable answer, and it is worth full marks. AI usage: you may ask a model to explain what a figure means. You may not ask it for the figure.

Rep 9 — Interrogate a TOPS claim

Find a real, published on-device AI performance claim — a vendor page, a product announcement, a spec sheet — that quotes a TOPS figure or an “AI performance” number. Do not use a model to find it; use a search engine and read the vendor’s own page.

Now apply §8.9’s four questions in writing, one paragraph each:

1. At what precision is the figure quoted? Does the page say? If it does not, note that.
2. At what utilization? Is any achieved-versus-peak information given anywhere?
3. Sustained for how long? Is there any statement of thermal conditions or measurement window?
4. With what memory system behind it? Is the memory configuration stated on the same page?

Then write the verdict in two sentences: what the claim does legitimately support, and what a reader would probably have concluded from it. Finally — and this is the rep — write the one additional sentence the vendor could have added that would have made the claim complete. Do not editorialize about the vendor; the point is the anatomy of an incomplete claim, not the character of anyone who made one.

Reps 10–11: Emerging Systems and the Log

Rep 10 — Rewrite the workload for four form factors

Pick one workload you can describe concretely: continuous speech transcription, a camera preview with real-time scene detection, a navigation session with the screen off, or token generation from a small on-device model.

Now write four short design sketches — wearable, phone, tablet, XR headset — each answering all Four Questions: performance, energy, thermals, placement. Each sketch is at most 120 words, and each must state:

• which processor the bulk of the work lands on, and why;
• what has to be given up at that form factor (not “it’s slower” — say exactly what capability you cut);
• the single constraint that dominates the design.

Then, in three sentences, name what changed most across the four and what changed least. If your four sketches differ only in scale, you have not done the rep — go back and find the place where the constraint changes the architecture, not just the numbers.

Rep 11 — Audit your own measurement log

Open the measurements.xlsx you have been keeping since Rep 1 and audit it as a hostile reviewer would. For every row, check:

• Is there a median and a dispersion figure? (A bare number fails.)
• Is the machine identified?
• Is the power state stated — mains or battery?
• Is the thermal condition stated — idle for how long, or which consecutive run?
• Is the repetition count and warm-up count stated?
• Is the compiler and optimization level stated for anything you built?
• Could a stranger with your machine reproduce this row from the log alone?

Fix every row that fails. Then write two sentences: which field you omitted most often, and what a reader could have wrongly concluded because of it. Then the honest one: find the row in your own log that most flatters a conclusion you were hoping for, and write the sentence that discloses it. That sentence is the professional skill this entire chapter exists to build, and the capstone rubric has a line for it.

Done? One Last Thing.

This is the capstone in miniature — one honest comparison, start to finish, in under a page.

Take two things you can genuinely compare: two builds of the same kernel, two --kernel selections from the harness, or the barriered and unbarriered binaries from Rep 2. Then produce, in this order:

1. The claim, in one sentence, with its metric named before its winner. (“Measured as median nanoseconds per operation over 31 repetitions, X is N% faster than Y.”)
2. The measurement, reported as median and IQR, with the full conditions block from Appendix C.
3. The mechanism — one paragraph saying why, tied to a specific idea from Chapters 1–7 by number, and to a counter or a structural argument if you have one.
4. The confounds — everything you did not control, listed plainly, including anything you could not measure.
5. The flattering choice — the single measurement decision you made that most helps your conclusion, named out loud, plus the number your claim becomes if you reverse it.
6. The AI-usage note, per Appendix D.

Six items, one page. Keep it — that is the exact skeleton of the capstone’s report.docx, and of the first three minutes of your recorded briefing. You have now written it once, on something small, where being wrong costs nothing.

Up next: Project 8 — Project 8: The Capstone SoC Investigation. Read the exam document the same day, so the three challenge questions shape your investigation instead of ambushing it.