Chapter 8 — Measure, Compare, Decide

“Measure, don’t guess.” — an engineering adage, attributed to the trade rather than to any one person

“A false balance is an abomination to the LORD, but a just weight is his delight.” — Proverbs 11:1 (ESV)

Why This Matters

For seven weeks you have been learning mechanisms. Exception levels and cache lines. Prime, performance and efficiency cores behind a shared coherency point. Tile memory, matrix engines, quantization, operator fallback. Voltage, frequency, leakage, thermal mass, and the skin-temperature limit set by a human hand. A modem that is a whole computer sitting next to yours. A root of trust fused into silicon.

This week the mechanisms have to become a claim. Somebody is going to ask you which part is faster, or where the bottleneck is, or whether the new model belongs on the NPU. Whatever you answer, you answer with a number — one you measured, one you read, or one a vendor, a benchmark suite, or a language model handed you with total confidence and no source at all.

Here is what makes that hard, and it is not the arithmetic. Every number you can produce about a mobile system is a number about a situation. Which cores. Which clock. Plugged in or on battery. Cold or soaked. Which build of the runtime. Which second of the run. Change one and the number changes — sometimes by more than the architectural difference you were trying to measure. Mobile makes this acute, because the thermal envelope means the same silicon legitimately produces two different numbers ninety seconds apart. A number without its conditions is not a small omission; it is a different claim than the one it appears to be.

So this chapter teaches three motions and grades all three. Measure — a defensible number: warmed up, repeated, reported as a median with dispersion under stated conditions. Compare — two systems side by side with the confounds named and, where possible, normalized away. Decide — the architectural judgment the measurement was for, which is the part no tool does for you.

Both of this book’s AI registers converge here. AI as the workload: on-device inference is the hardest thing in this course to benchmark honestly — memory-bound, thermally limited, precision-dependent, and quietly falling back to the CPU for one operator mid-graph. Every error in this chapter compounds in an inference benchmark. AI as an untrustworthy research assistant: ask a model for a cache latency, a bus width, a TOPS rating, and you get a specific, plausible, well-formatted figure. It may be right; it has no way to tell you which. Use it to explain, never to source. And the week’s Christian question is not decorative, because Proverbs 11:1 is about a merchant’s stone, not a metaphor. A measuring instrument sits exactly where knowledge becomes trust. If the instrument is crooked, everyone downstream reasons correctly to a false conclusion and never knows. We come back to it in §8.13 — by then you will have watched yourself make the same silicon look fast or slow using nothing but defensible choices.

8.1 — What a Number Has to Carry

A measurement is not a number. A measurement is a number plus everything required to reproduce it. Six requirements, each of which exists because skipping it produces a specific, predictable lie.

1. Warm up, and throw the warm-up away. The first iteration of anything measures cold caches, an untrained branch predictor, unfaulted pages, and a core still at a low DVFS operating point because the governor has not noticed you exist.

2. Run to steady state — and know which one. There are two, and they are different claims. Microarchitectural steady state arrives in milliseconds: caches warm, predictor trained, clocks boosted. Thermal steady state arrives in minutes, and it is lower. Chapter 5 taught you why. Say which one your claim is about.

3. Repeat, and report a median with dispersion. A single number is an anecdote with a decimal point. Run an odd number of repetitions — 31 is a good default for a fast kernel — sort, and report the median plus the interquartile range. The median resists the one run where a background process woke up; the IQR is the part everyone omits and the part that tells a reader whether to believe you. If the IQR is comparable to the difference you are claiming, you have not measured the difference.

4. State the conditions. Machine, OS build, compiler and flags, power source, thermal condition, core placement, what else was running. Appendix C calls this the measurement log, gives you the template, and requires it in every lab. It is the difference between a result and a rumor.

5. Defeat the optimizer. At -O2 a compiler may notice you never use the result of your loop and delete the loop. You then measure, with impressive precision, the cost of nothing.

6. Compare only like against like. Never put a cold-device number next to a soaked one, or a plugged-in number next to a battery number, and call the difference architecture. That is the false balance, and it is usually an accident.

The harness that does all of this is code/bench_harness.cpp — small enough to read in one sitting, because you will reuse it for the capstone and should not trust an instrument you have not read.

g++ -O2 -std=c++17 -o bench_harness code/bench_harness.cpp
./bench_harness --reps 31 --warmup 3 > bench.csv

The barrier is two lines and worth understanding completely:

static volatile std::uint64_t g_sink = 0;
static inline void do_not_optimize(std::uint64_t v) { g_sink ^= v; }

A write to a volatile object is an observable side effect the compiler may not remove, so the value handed in must actually be computed. That is the whole trick. Production harnesses use cheaper inline-assembly barriers; this one is portable, obvious, and correct.

Here is one real run, captured on the laptop this chapter was written on, at 2²⁰ elements. Your numbers will differ, and that is the point — read the shape:

kernel,elements,ops_per_rep,reps,median_ns,iqr_ns,min_ns,max_ns,ns_per_op
alu,1048576,1048576,31,992292.0,34604.0,975542.0,1153875.0,0.9463
seq,1048576,1048576,31,79209.0,3334.0,76500.0,92500.0,0.0755
stride,1048576,131072,31,72125.0,2042.0,70209.0,82041.0,0.5503
chase,1048576,1048576,31,8400459.0,1372458.5,7206333.0,14057125.0,8.0113

Three things in that block matter more than the values.

• stride costs about seven times seq per access. It touches every eighth 8-byte element — one access per 64-byte cache line — so it pays a whole line’s transfer for one useful word. Chapter 3’s cache-line lesson, reproduced on your own machine in under a second.
• chase costs roughly a hundred times seq per access. Same array, same loads, same instructions per step. The only difference is that each load’s address depends on the previous load’s result, so the prefetcher cannot run ahead and the core cannot overlap. That is dependent-load latency with bandwidth and memory-level parallelism stripped away.
• alu — the kernel with no memory traffic at all — is slower per operation than seq. Nearly everyone predicts otherwise. The multiply-add chain is dependent, so it runs at multiplier latency with no overlap, while the summation loop is independent work the core can pipeline and vectorize. The lesson is not “compute is slow.” It is that “compute versus memory” is the wrong axis; the right axis is dependence.

Note also that min sits well below median and max far above it. Had you run once, you would have reported anything in that spread — honestly, and wrong by tens of percent.

Coach’s Note — Report the median, not the minimum. There is a school of benchmarking that reports best-of-N on the theory that the fastest run is the “true” one with the noise removed. That is defensible on a fixed-clock machine with nothing else running. On a phone it is nonsense, because the fastest run is the coldest run, and coldness is not a property of the silicon — it is a property of the last four minutes. Best-of-N on a thermally limited device measures your patience between runs.

8.2 — Microbenchmark, Application Benchmark, and What Each Can Support

Both kinds of run are legitimate, and each supports a different kind of sentence. Confusing them is the second most common analysis error in this field.

A microbenchmark isolates one mechanism. chase above measures dependent-load latency and essentially nothing else. Its virtue is that it is unconfounded: when the number moves, you know what moved it. Its vice is that it is trivially easy to make precisely wrong — a working set that accidentally fits in L2, a stride that lines up with the cache’s associativity, a compiler that vectorized the loop you thought was scalar. It supports sentences of the form “this mechanism, in isolation, costs about X” — and no sentence at all about how long an app takes.

An application benchmark runs the real thing: a frame of the actual game, a token from the actual model, the actual camera pipeline. Its virtue is representativeness; its vice is confounding — the number moved, and it could have been memory, the scheduler, thermal state, a garbage collection, or a driver update. It supports “under these conditions, the user waits about X,” and nothing about why until you attach counters.

You need both, and you must say which produced each number; the capstone rubric checks this. The professional shape is: application benchmark to find that something is slow and how slow it feels; counters to hypothesize why; microbenchmark to confirm the mechanism in isolation; application benchmark again to confirm the fix moved a number a person can feel.

Coach’s Note — The most expensive mistake in performance work is optimizing a microbenchmark and shipping the result. You make the isolated kernel four times faster and the application zero percent faster, because the kernel was two percent of the frame. Before you tune anything, get its share of the whole — a speedup bounded by Amdahl’s ceiling is a true number about an irrelevant situation.

8.3 — Utilization Is Not Performance

If you take one sentence from this chapter into your professional life, take this one: a core reported at 100% utilization may be doing nothing at all.

Utilization, as every operating system reports it, is a time measure: the fraction of wall-clock time the scheduler had a runnable thread on that core rather than parking it in an idle state. It is not a measure of work. A core stalled for the hundreds of cycles a DRAM round trip typically takes is not idle — it is occupied, waiting — and every one of those cycles counts as busy. Chapter 3 taught you how often that happens; Chapter 5 taught you that those cycles still cost leakage and clock power.

The confusion drives an enormous amount of bad engineering. “The CPU is pegged, so we need faster cores” — when the fix is a data-layout change that cuts the miss rate, and a faster core would have stalled just as long in nanoseconds. “The NPU shows activity, so acceleration is working” — when it is running four of your seven operators and bouncing to the CPU for the other three.

The fix is a discipline, not a tool. Pair every utilization number with an IPC and a miss rate. Three numbers together tell a story none of them tells alone:

That table is the analysis half of this course compressed into eight lines. Learn it.

8.4 — The Counters That Matter, and Where They Lie

Arm cores implement a Performance Monitor Unit — hardware counters that increment on real microarchitectural events. Other architectures have their equivalents. These are not estimates and not samples; they are counts of things that physically happened.

Two derived quantities carry most of the diagnostic weight.

IPC = instructions ÷ cycles. The best first look. Interpret it relative to the core’s issue width, which is why “good IPC” is not a fixed number: 1.4 might be excellent on a narrow efficiency core and disappointing on a wide prime core running the same binary. Chapter 2’s point — same ISA, different microarchitecture — is exactly why you cannot read IPC without knowing which core you were on.

MPKI = misses ÷ (instructions ÷ 1000). Prefer MPKI to a miss rate when comparing two versions of the same program, because a rate over accesses can improve while absolute stall time gets worse. MPKI normalizes against useful work.

code/counters.py reads perf stat- and simpleperf stat-shaped output, does this arithmetic, and prints a documented first-pass diagnosis. Sample counter output ships as code/sample-perf-stat.txt — synthetic, and labeled as such on its first line — so Workbench B students with no counter access can still do every counter rep this week.

python3 code/counters.py code/sample-perf-stat.txt

run: ./matmul_ijk
  instructions     19,905,331,204   cycles     25,203,884,112
  IPC     0.79     L1D MPKI    37.61     LLC MPKI     8.08
  dTLB MPKI   0.93     branch miss    0.50 %
  DIAGNOSIS: MEMORY-BOUND. Last-level misses dominate: every one is a DRAM
  round trip. Look at working-set size and access pattern (Ch 3), then at
  whether this work belongs on another processor (Ch 4).

On a Linux box with counter access you produce your own:

perf stat -e cycles,instructions,branches,branch-misses,\
L1-dcache-loads,L1-dcache-load-misses,LLC-loads,LLC-load-misses \
  ./bench_harness --kernel chase 2> run.txt
python3 code/counters.py run.txt

On Android with a debuggable build, simpleperf stat is the equivalent and parses with the same tool. On macOS the counter path is Instruments and its driver xctrace; the format differs, so enter the numbers by hand rather than pretending you ran something you did not. On many Linux systems unprivileged counter access is restricted by a kernel setting and must be relaxed by an administrator.

Now the part everyone skips, which is why everyone eventually publishes a wrong number. Counter multiplexing. A PMU has a small, fixed number of physical counters. Ask for more events than there are counters and the kernel does not refuse; it time-slices them, running each for part of the interval and scaling the result up to an estimate. perf prints the percentage next to each event, and almost nobody reads it. So: a multiplexed count is an extrapolation whose error grows as behavior varies across the run, and two events multiplexed onto different slices were not observed simultaneously, so a ratio computed from them can be nonsense even when both counts are individually near-correct. Ask for a small event set that fits, measure, then run again with the next set. Three clean runs beat one run with twelve multiplexed events.

Coach’s Note — When a counter number surprises you, your first hypothesis should be that you measured the wrong thing, not that you discovered something. Was the event multiplexed? Did the process migrate to a different core class mid-run, averaging two microarchitectures? Did you count start-up when you meant the kernel? Extraordinary counter values almost always mean an ordinary methodology bug, and suspecting yourself first is most of what separates a measurement from a rumor.

8.5 — GPU and NPU Utilization: The Same Error, Only Worse

Everything in §8.3 applies to the accelerators, then gets worse, for three reasons.

“GPU busy” is measured differently and means less. A mobile GPU utilization figure typically reports the fraction of time the GPU had work queued and was not idle. It does not tell you whether shader cores were issuing, whether they were waiting on texture fetches, or whether a handful of threads in a group had diverged and serialized the rest — Chapter 4’s SIMT divergence, invisible in the summary. A GPU at “90%” that is bandwidth-starved and one that is arithmetic-limited want opposite fixes.

On a tile-based GPU the expensive thing is often not on the chart at all. Chapter 4’s central mobile-graphics lesson was that tile-based rendering exists to keep framebuffer traffic off DRAM, and that an unnecessary render-target load or a gratuitous framebuffer read-back forces tile memory out to main memory. That cost appears as memory traffic and energy, not as GPU busy time — so you can make a frame dramatically more expensive with a change that barely moves utilization. The counter you want is bandwidth, not occupancy.

The NPU is the least legible processor on the chip. Vendors expose wildly different visibility, and the number that matters most is one the accelerator has no reason to advertise: which operators actually ran on it. Chapter 4 named operator fallback as the classic disappointment — one unsupported operator mid-graph, a round trip back to the CPU, and an “accelerated” path slower than the CPU path it replaced, while an activity meter cheerfully shows the NPU doing something. As of 2026, NNAPI is deprecated (in Android 15, API level 35), and the landscape is framework delegates and vendor SDKs, still moving and still vendor-specific. So: do not infer placement from a utilization graph. Get the runtime to report per-operator assignment, and if it will not, measure the end-to-end latency of the CPU-only path as your control.

When the workload is a language model, add Chapter 3’s arithmetic first. Token generation on-device is typically memory-bound — the binding constraint is moving weights and a growing key-value cache, not multiply-accumulate throughput — so a benchmark reporting only compute utilization is measuring the part that was never the problem.

8.6 — Peak, Sustained, and the Honesty Problem

Chapter 5 is the chapter this section owes everything to. A phone has no fan; its sustained dissipation is bounded by conduction into a chassis and by a skin-temperature limit set by what a human hand can comfortably hold — not by what the silicon can survive. But the chassis has thermal mass, so for tens of seconds the device absorbs a burst far above what it can sustain. That is not a defect; it is the correct design for a device whose real workloads are bursty. It does mean that peak and sustained are two different true claims about the same silicon, and quoting one while your reader assumes the other is the purest false balance available in our field.

1. Quote both, or quote neither. A headline number without a sustained companion loses rubric points, in the labs and in the capstone.
2. A single run on a cold device measures the chassis — a real number about the last four minutes of idleness, not about the architecture. And never compare across differently conditioned devices: one out of a pocket and one off a desk is a temperature experiment with a benchmark attached.
3. Cooldown between runs is a measurement choice, and the most powerful knob you have for making silicon look good. Five minutes of cooldown between five-minute runs produces a very different chart than back-to-back runs, and both are honest — as long as you say which you did.
4. Report the sustained/peak ratio as a first-class result. It is a compact statement about a design’s thermal honesty, often more informative than either raw score.

Chapter 5 also gave you the platform side: Android exposes a thermal status ladder — THERMAL_STATUS_NONE, LIGHT, MODERATE, SEVERE, CRITICAL, EMERGENCY, SHUTDOWN — through PowerManager.getCurrentThermalStatus(), and a forecast through getThermalHeadroom(int forecastSeconds), normalized so 1.0 is the throttling threshold. Log those next to your scores when you can. A chart showing the score falling and the status climbing has explained itself; one showing only the falling score has reported a mystery.

You do not need a phone for any of this — laptops throttle too, and Chapter 5’s soak protocol in Appendix C works on any machine. The capstone requires a stated thermal condition for every reported number, and does not care whether it is “plugged in, idle 10 minutes, ambient ~22 °C” or “third consecutive five-minute run.” It cares that you said.

8.7 — Measuring Power When You Have No Power Meter

Every question in this course has an energy half, and energy is the one thing an ordinary student workbench cannot measure directly. Be honest about that rather than inventing a number — then get as far as honest reasoning takes you, which is further than most people think.

What you generally cannot do without instrumentation: attribute watts to a specific block on the SoC. Package power, where exposed at all, is a whole-chip figure, and the split among CPU cluster, GPU, NPU, memory controller and display is not something you can back out from software. Do not claim it.

What you often can do:

• Energy per task at a fixed operating point. E = P × t. If you cannot get P, note that for a fixed workload on a fixed configuration time is proportional to energy — so measured time is a legitimate relative energy proxy, and say exactly that when you use it. Two implementations of one kernel, same machine, same power state: the faster used less energy. Defensible.
• Battery-delta measurement. Run a long, steady workload between two known charge levels with screen and radio state controlled, then convert with Chapter 5’s exact arithmetic: watt-hours = mAh × nominal volts ÷ 1000. Crude, but real, if you state run length and device state.
• Platform energy counters where they exist, or proxy by exclusion — idle draw over a window versus the same window with your workload. Either way, name what the number includes precisely, and note that a difference attributes everything that changed, not just your code.

What you must never do: multiply a benchmark score by a plausible-looking wattage you did not measure and present the quotient as performance-per-watt. That is a fabricated figure with arithmetic performed on it, which makes it worse than a guess, not better — it now looks derived.

Coach’s Note — “I could not measure this, and here is exactly what I would need in order to” is a strong sentence in an engineering report, not a weak one. It tells your reader you know the boundary of your evidence, which is the only reason to trust anything inside it. The capstone has a required section for precisely this, and students routinely find it is the section a reviewer respects most.

8.8 — The Tool Landscape, Named Correctly

You will be asked, in an interview or a design review, what you profile with. Know the real names and what each is for.

The Jetpack benchmark libraries repay study even if you never ship Android code: they encode this chapter’s rules — warm-up, repetition, thermal-throttle detection, refusal to run on a debuggable build — as enforced behavior rather than advice.

On Workbench B — the browser-based, no-install, no-admin path every Normal-tier requirement must be completable on — you will not have counter access. That is planned for: the portable C++ harness runs, the Python analysis runs, and the counter reps run against the shipped sample. The capstone’s required measurements are timing measurements, and counter analysis may be done on the supplied sample with the limitation stated.

8.9 — Comparing Real Silicon Honestly

Two parts, two spec sheets, one question: which is better?

The professional answer begins by refusing the question as asked. “Better” is not a property of silicon; it is a relation between silicon and a workload under a constraint. The useful question is always some version of: for this workload, under this power and thermal envelope, at this cost, which architectural choices help and which hurt?

What you have to normalize first

A cross-part comparison is a controlled experiment in which you control almost nothing. Name each confound; normalize what you can; report the rest as limitations.

Do the bandwidth line by hand every time, because it is arithmetic rather than a claim and it frequently explains a result people attribute to the CPU:

bytes/s = data rate (MT/s) × bus width (bytes)

A 64-bit — that is, 8-byte — interface at 8533 MT/s gives 8533 × 10⁶ × 8 ≈ 68 GB/s. Double the width to 128 bits at the same data rate and you have doubled the bandwidth without touching the DRAM generation. As Chapter 3 put it: the width, not the generation, is usually what bought the bandwidth. Phones commonly use a 64-bit-class interface; tablet- and laptop-class parts often use much wider ones. Check the specific part’s specification; do not assume.

The single-number score, and TOPS

A benchmark suite produces one number by combining several workloads under a weighting somebody chose. That weighting is an editorial judgment about what matters, presented as a measurement. It is not dishonest — it is often carefully documented — but by the time it reaches a chart it has discarded which workloads it contains, how they were weighted, whether they were memory- or compute-bound, at what thermal state, over what window, on which cores. Read the suite’s own methodology page before you cite the suite’s own number.

“Tera-operations per second” deserves specific treatment, because it has become the headline number for on-device AI. A TOPS figure is a peak theoretical rate: multiply-accumulate units × operations per unit per cycle × clock. It is arithmetic about the hardware’s shape, not a measurement of anything running. Four questions dismantle any TOPS claim:

1. At what precision? INT4, INT8, FP16 and FP8 rates all differ, and quoting the smallest-datatype figure is standard practice. A number without a datatype is not a number.
2. At what utilization? Real graphs do not keep a matrix engine fed, and the gap between peak and achieved is workload-dependent and often large.
3. Sustained for how long? It is a peak rate on a device with a thermal envelope. Everything in §8.6 applies.
4. With what memory system behind it? On-device inference is typically memory-bound. Doubling the multiply-accumulate array on an unchanged memory system can buy very little.

Ask those four out loud in a review and you will be the most useful person in the room. TOPS is not a lie; it is an incomplete claim presented in a context that invites a complete reading. That is precisely what §8.13 is about.

Four parts, five metrics, four different winners

Shipped with this chapter is code/soc-compare.csv.

Read this carefully, because it is a design decision and it is graded. The four parts in that file are named SoC-A, SoC-B, SoC-C and SoC-D, and they are entirely fictional — not code names, not stand-ins, not thinly disguised versions of any shipping product from any vendor. They were constructed precisely so that this exercise can be run without implying a single claim about anybody’s real silicon, which is the kind of claim this book refuses to make: ranking named vendors’ parts against each other is not architecture, it is sport. The file’s first line declares it synthetic teaching data, as every authored dataset in this book must. In your capstone the data will be real, the citations primary, and the same honesty rule applies.

python3 code/compare.py

soc     cpu        gpu            GB/s       peak sustained    watts  sus/peak
SoC-A   1P+3M+4E   wide           68.3       4200      2600      6.4     0.619
SoC-B   4M+4E      medium         60.0       3600      3100      5.0     0.861
SoC-C   2P+4M+4E   very-wide     120.0       4600      3900      9.2     0.848
SoC-D   2M+6E      narrow         51.2       2400      2200      2.6     0.917

Ranking by each metric, best first:
  peak score:              SoC-C  >  SoC-A  >  SoC-B  >  SoC-D
  sustained score:         SoC-C  >  SoC-B  >  SoC-A  >  SoC-D
  sustained/peak ratio:    SoC-D  >  SoC-B  >  SoC-C  >  SoC-A
  sustained perf per watt: SoC-D  >  SoC-B  >  SoC-C  >  SoC-A
  DRAM bandwidth (GB/s):   SoC-C  >  SoC-A  >  SoC-B  >  SoC-D

Five metrics from one honest dataset, and the winner is not the same part twice running. SoC-C leads on peak, sustained and bandwidth — and places third on both efficiency metrics. SoC-D is last on every absolute measure and first on both ratios. SoC-A is second on peak and dead last on perf-per-watt: a design that spends its thermal budget early, defensible for a bursty product and terrible for a sustained one.

Nobody in that table lied. Somebody chose a metric. The tool’s closing line is the sentence I want you able to say in a meeting:

  Nobody lied. Somebody chose the metric. Say which one you chose,
  and say it before you say who won.

And the architecture explains the numbers once you look. SoC-C has twice the memory bus width of the others — that is where its lead on any bandwidth-sensitive workload comes from — and the highest sustained power, which is a form-factor statement, not a virtue. SoC-D has no prime core and a narrow GPU: exactly what you would design for a product with a fraction of the thermal path. That is the level at which architects compare parts — not “which won,” but “what was each one optimizing, and does it match my product?”

Coach’s Note — When somebody hands you a comparison, ask one question before any other: who chose the metric, and what were they optimizing for when they chose it? It is not an accusation; it is the same question you should ask of your own charts. The honest form of a comparison names its metric first and its winner second. Reverse that order and you have written an advertisement, whatever the numbers say.

8.10 — Where This Goes Next

This is the part of the course with the shortest shelf life, so it is also where hedging is a professional skill rather than a dodge. Here is the landscape as of 2026, at the level I am confident in.

Form-factor divergence. The same architectural vocabulary now spans wildly different budgets, and the budget — not the vocabulary — determines the design. A wearable has a fraction of a watt and millimeters of thermal path; almost everything is offloaded, duty-cycled, or handled by a sensor-hub-class always-on processor (Chapter 6), and sustained anything is essentially unavailable. A phone sustains on the order of a few watts, bursts well above it, and is the design centre of this book. A tablet has more area, more chassis mass and a bigger battery, so the same ideas run at a higher sustained point — the clearest demonstration in consumer hardware that the chassis is part of the architecture. An XR headset carries the hardest constraint in consumer computing: a hard, low motion-to-photon latency budget, widely stated as a design target in the low tens of milliseconds, under a thermal and weight limit sitting on a human face — which is why some products split work between the headset and a tethered puck or phone. Take one workload, place it on all four: the instruction set does not change; the answer changes completely.

Edge AI. The drivers are clear and durable: latency, privacy, offline operation, per-query cost. So are the limits: memory footprint, sustained thermals, model quality at low precision — Chapters 3, 5 and 4 respectively, which is a good sign this book taught the right things. Everything else in edge AI moves fast enough that you should trust your own measurement over any document, including this one.

RISC-V application processors. The vector extension version 1.0 was ratified in November 2021, and the RVA23 application-processor profile (ratified 2024) makes vector mandatory for compliant application processors — a clear statement of intent about data-parallel and machine-learning work. As of 2026, RISC-V ships in volume in embedded and accelerator roles, including as control cores inside SoCs that are otherwise Arm. It is not the application-processor ISA of mainstream phones, and I will not predict when or whether that changes. Watch it for the licensing and customization model, not a claimed ISA advantage — Chapter 1 was clear that the instruction set is a real but second-order effect.

Chiplets and advanced packaging. Disaggregating one large die into several improves yield and lets each die use the process best suited to it; the costs are inter-die communication energy and packaging complexity. UCIe is the industry standard effort for die-to-die interconnect; the vocabulary to know is 2.5D interposers, 3D stacking, package-on-package DRAM. These techniques are further along in data-centre parts than in phone SoCs — be careful with adoption claims.

The question to leave with. Chapter 4 argued that specialization is the only remaining way to buy performance when you cannot buy watts, and two decades of silicon agree. So — where does the accelerator fleet stop? Every fixed-function block is silicon that is dark for most workloads: enormous efficiency for the workload it was built for, nothing at all for the one that arrives next year. Generality has a value that never appears on a spec sheet — being adequate at the thing nobody anticipated. A chip that is a fleet of perfect accelerators for 2026’s workloads ages badly; a chip that is all general-purpose cores cannot meet 2026’s energy budget. Every real design is a bet placed between those, under uncertainty, years before the workload arrives. That bet is architecture. No benchmark will make it for you.

8.11 — Closing the Loops: What Each Week Contributed

The capstone is not a new assignment. It is the assembly of seven weeks, and the rubric grades whether you used what each one gave you.

Chapter 1 — The Machine in Your Pocket. The Four Questions, block-diagram literacy, and the ISA foundations — AArch64 registers and exception levels, the honest RISC-versus-CISC story, vector-length-agnostic SIMD. It started your soc-architecture-review.docx, the document the capstone finishes. Every bottleneck prediction you make this week is that Week 1 skill with measurements attached.

Chapter 2 — Not All Cores Are Equal. Why a measurement is meaningless without knowing which core it ran on: same ISA, different microarchitecture, different IPC, different energy — plus migration cost, affinity versus quality-of-service classes, and performance-per-watt. When §8.4 said you cannot interpret IPC without the core class, Chapter 2 was collecting a debt.

Chapter 3 — The Real Bottleneck. The hierarchy, the cache line, locality, bandwidth-versus-latency, the bandwidth arithmetic you used in §8.9, and the headline energy fact — Horowitz’s ISSCC 2014 figures, at 45 nm and to order of magnitude, showing a DRAM access costing three to four orders of magnitude more energy than the arithmetic on the datum. Most of what your counters say this week is a Chapter 3 story.

Chapter 4 — Beyond the CPU. The placement question and the placement table, tile-based rendering and the framebuffer round trip, quantization arithmetic, and operator fallback — the trap §8.5 told you to test with a CPU-only control. Its Week 4 Placement Practical is what the capstone’s placement table extends.

Chapter 5 — The Budget That Governs Everything. The hinge of the course and the parent of §8.6 and §8.7: the efficiency knee, race-to-idle and its limits, energy as the currency, the fan-less thermal path and the skin-temperature limit, thermal mass, and the platform thermal ladder. Every sustained-versus-peak sentence you write this week is Chapter 5’s.

Chapter 6 — Wiring the System Together. The interconnect as the real system architecture, quality-of-service arbitration and the display controller that cannot be starved, interrupts versus polling as an energy decision, and the radio lesson that batching beats compressing. When the bottleneck is not the CPU, the memory or the accelerator, Chapter 6 is where it is.

Chapter 7 — Trust in Silicon. The root of trust, isolation by mode versus isolation by silicon, hardware-backed keys and attestation, and the law that a performance optimization is a potential side channel — with the data-memory-dependent-prefetcher work (Augury, 2022; GoFetch, 2024) as the modern example. Chapter 7 promised that mitigation costs performance and that Chapter 8 would measure it; the capstone’s Medium tier asks you to do exactly that, then decide in writing whether you would pay it.

Seven chapters, one document, four questions. Now go finish it.

8.12 — Interactive Lab: The Honest Benchmark

Below this chapter on the website you will find The Honest Benchmark. Use it before you start the capstone; it is the fastest way to feel §8.1, §8.6 and §8.9 in your hands rather than in your notes.

Panel one hands you a claim — “SoC-A is 30% faster than SoC-B” — and then hands you the measurement conditions: run length, cooldown between runs, repetition count, plugged in or on battery, which cores, which metric gets reported. Every one is individually defensible; several are what a careful engineer would do. Move them and watch the number move. Your goal is deliberate and uncomfortable: make the same silicon look as good as you can, then as bad as you can, without ever making a choice you could not defend out loud. The verdict panel names which configurations crossed from defensible into false balance, and the reason is never “you used a wrong number.” It is always “you reported a number whose conditions your reader would not have assumed.”

Panel two is the diagnostic drill: a counter set — IPC, L1 and last-level miss rates, branch miss rate, stalls — and the question what is the bottleneck? It grades you and explains the misses. Run it until you are right without hesitating, because in the capstone you get counter sets with no answer key, and in your career you get them with a deadline.

What the lab teaches, and what a widget cannot say as bluntly as I can: the dangerous benchmark is not the fraudulent one. It is the one where every individual choice was reasonable and the aggregate is misleading. Fraud is rare and easy to condemn. This is common, easy to commit by accident, and why this chapter exists.

8.13 — A Just Weight

Proverbs 11:1 is about a stone. “A false balance is an abomination to the LORD, but a just weight is his delight” (ESV). In the ancient Near East a merchant carried weights in a pouch and a balance beam over his shoulder, and the buyer could audit neither. A dishonest merchant did not need to lie; he needed only two sets of stones — heavier for buying, lighter for selling — and let the instrument lie for him. Everyone downstream then reasoned perfectly correctly, from a corrupt input, to a false conclusion, and never knew.

That is why the language is so strong. Ordinary dishonesty deceives a person; a crooked instrument deceives a system. It corrupts the shared basis on which people who cannot verify each other’s claims nevertheless manage to trade — and once you cannot trust the scale, you cannot trust anything weighed on it, including the honest transactions. The false balance is called an abomination not because it is the worst sin available but because it is load-bearing: it poisons the mechanism by which truth normally circulates.

Now look at what you have done all week. A benchmark is a balance; a performance counter is a weight; a comparison chart is a public market in which people who cannot run the experiment decide what to believe and what to build on. When you publish a number you hand someone a stone and ask them to trust the pouch it came from.

Here is the sharp edge, and it is why this section sits at the end of this chapter rather than in an ethics unit: a technically true number presented without its conditions is a false balance. You fabricated nothing. Every digit is real. And yet if you quoted a peak number to a reader who assumed sustained, or a plugged-in number to one who assumed battery, or a composite score to one who assumed it represented their workload — you handed over the lighter stone and let the instrument lie on your behalf. You will meet this in your first job, and it will not arrive as a request to falsify anything. It will arrive as a chart due Thursday, a number that came out badly, and a completely legitimate methodological choice that would make it come out better — one you can defend, that nobody will catch, because nobody can rerun it. The answer is not a rule; it is a habit built now, where being wrong costs nothing: state the conditions before you state the number, and name the choice that most flatters your conclusion out loud, before someone else finds it.

There is a positive claim in the proverb too, easy to skip: a just weight is his delight. Not merely permitted — delighted in. There is a strand in Christian thinking, and the Lutheran tradition presses it hard, that ordinary competent work done honestly is not spiritually neutral filler between the important things; it is the important thing, for the person called to it. The one who keeps an honest scale in a market nobody is auditing is doing something God is said to take pleasure in — and notice that the market cannot tell. The honesty of a measurement is almost always invisible to those relying on it, which is precisely why it must be a matter of character rather than of enforcement. There is a warning here for the AI age too. A language model will hand you a beautifully formatted, entirely fabricated cache latency with no more hesitation than a correct one, because it has no scale at all — no instrument, no measurement, no way to distinguish the weight it remembers from the weight it generated. It is not lying; lying requires knowing. It is a merchant with a pouch of stones of unknown provenance, offering you one with complete confidence. Use it to explain, never to source. The weighing stays with you — and that is not a limitation of today’s models you are waiting out. It is what it means to be the one holding the balance.

8.14 — Common Pitfalls

Pitfall: Reporting a single run, or a best-of-N, as a result. Example: “The optimized kernel takes 68 ms” — from one run, on a cold machine, with no dispersion. Fix: Warm up and discard, run an odd number of repetitions (31 is a good default), report the median and the IQR. If the IQR is comparable to the difference you are claiming, you have not measured the difference.

Pitfall: Reading utilization as performance. Example: “Both cores are at 100%, so we are compute-bound and need a faster CPU.” Fix: Pair every utilization figure with IPC and a miss rate. High utilization with low IPC and high last-level MPKI is a memory problem (Chapter 3); a faster core stalls just as long. Use §8.3’s table first.

Pitfall: Quoting a peak number to a reader who will assume it is sustained. Example: A chart headed “performance,” built from 60-second runs on devices that had been idle ten minutes. Fix: Quote both or quote neither, and report the sustained/peak ratio as a first-class result. Label the measurement window and the thermal condition on the chart itself, not in a footnote nobody reads.

Pitfall: Trusting a counter set that was multiplexed. Example: Asking perf for twelve events at once, then computing a miss rate from two that were never observed in the same time slice. Fix: Read the multiplexing percentage perf prints next to each event. Ask for a small event set that fits, measure, then run again with the next set. A ratio built from two extrapolated counts is not a ratio.

Pitfall: Letting the compiler delete the thing you meant to measure. Example: A -O2 loop whose result is never used, timing at a suspiciously round few nanoseconds regardless of problem size. Fix: Use the barrier in code/bench_harness.cpp — consume every result through a volatile sink. Sanity-check by doubling the input size: if the time does not roughly double for a linear kernel, the loop is gone.

Pitfall: Attributing a cross-part difference to microarchitecture when the memory system, the process node, or the software stack explains it. Example: “Part X’s CPU is stronger” — on a memory-bound workload, where part X has twice the DRAM bus width. Fix: Do the bandwidth arithmetic (MT/s × bus bytes) for both parts before writing a sentence about cores. Then pin and state every software version; on accelerator paths the runtime and delegate version is frequently a larger effect than the silicon.

Pitfall: Sourcing a figure from a language model. Example: A capstone citing a last-level cache size or TOPS rating that appears on no vendor page, produced by a model that sounded certain. Fix: Every external figure in this course carries a primary-source citation — a vendor specification page, an architecture reference manual, or your own measurement. An uncited figure scores zero on its line; a fabricated one fails the integrity line outright.

8.15 — Reps

Open the exercises and do all eleven. This week’s reps are the measurement discipline: building and reading the harness, breaking it on purpose so you can recognize a broken one in the wild, diagnosing counter sets cold, and ranking four parts four different ways until “which is faster” stops sounding like a question with an answer.

This week’s AI policy: unchanged and enforced hardest. Use a model to explain a mechanism, a tool flag, or a statistic. Never use one to supply a hardware figure. Every rep involving a number about real silicon requires a primary-source citation, and every rep ends with the honest one-line AI-usage note Appendix D defines. In a course about honest measurement, an unsourced number is not a small error; it is the error the whole week is about.

A preview:

• Rep 2 — Delete the optimizer barrier and watch the benchmark lie about a loop that no longer exists.
• Rep 4 — Diagnose three counter sets cold, writing your verdict before the tool prints its diagnosis.
• Rep 5 — Utilization versus IPC: find the busy core that is not working.
• Rep 7 — Rank four SoCs four different ways and watch the winner change with the metric.
• Rep 9 — Interrogate a TOPS claim with the four questions until it survives or collapses.

A short “Check Your Reps” quiz is embedded on this page below the lab. It is an ungraded self-check; take it before you start the capstone. Then sit this week’s graded knowledge check in Canvas — worth 1.5%, same material, larger pool, randomized draw, so retaking it is real practice rather than memorization. It is the last of the eight.

8.16 — This Week’s Project

Two deliverables close the course, and together they are 28% of your grade.

Project 8 — The Capstone SoC Investigation, worth 20%. Pick one of three tracks: a comparison of two real, currently-shipping SoCs from different vendors on architecture rather than score, for one named workload; a workload study profiling one workload across CPU, GPU and NPU with a defended placement; or a design — a hypothetical SoC for a named product against a stated power and thermal envelope. All three require the same core: the soc-architecture-review.docx you started in Week 1, carried forward and finished; real measurements you took yourself, reported per Appendix C with medians and dispersion; a placement table in placement-decision.docx; a bottleneck analysis answering all Four Questions explicitly; primary-source citations for every external figure; and an honest section naming what you could not measure.

The Technical Briefing, worth 8%. A 12-minute recorded briefing defending your investigation to a skeptical engineering audience, plus a 3-minute addendum answering three challenge questions printed in the document. You get them in advance, and preparing for them improves the investigation itself.

Start the capstone the day you finish this chapter. Every requirement is something you have already done once in a lab; the work is assembling it into an argument, and arguments take longer to make short than to make long.

8.17 — Coach’s Final Word

Eight weeks ago I told you that a desktop architect asks how fast it can go, and a mobile architect asks four questions at once: how fast, at what energy, for how long before it throttles, and on which processor. Everything since has elaborated that sentence. The efficiency core is an energy answer. The tile is a bandwidth answer. The NPU is a data-movement answer. The thermal ladder is an admission that the platform cannot solve it alone. Not one of those exists on a machine with a wall socket and a fan.

This week you learned the thing that makes the other seven usable by anyone but you: how to say what you know in a way that survives being checked. A measurement with its conditions. A comparison with its confounds named. A recommendation with its evidence attached and its limits stated out loud. That is not a soft skill bolted onto the engineering. It is the engineering, at the moment it stops being private and becomes something another person will act on. And you learned where the temptation lives — not in fraud, but in the completely defensible choice that happens to flatter the answer, made under a deadline, on an experiment nobody else will rerun. A just weight is not a rule you follow when someone is watching; it is what you are when nobody is. Carry one set of stones. Say the conditions before you say the number. Name the choice that helped you before somebody else finds it.

You came into this course able to describe the components of a phone. You leave able to look at a block diagram and a specification table and say why the architect made those choices, where this workload will bottleneck, and how it should be mapped onto the hardware — with measurements behind every claim. That is a different profession from the one you started in. Go finish the capstone, record the briefing, and then go be the person in the room who asks: at what precision, at what utilization, and sustained for how long?

It has been a genuine pleasure. See you in the next course.

Up next: Do every rep in the exercises — the last conditioning session before the capstone. Then open Project 8 and pick your track, and read the exam document the same day so the three challenge questions shape the investigation instead of ambushing it. Appendix C has the measurement templates, statistics and soak protocol; Appendix B documents every dataset, including code/soc-compare.csv; Appendix A covers the workbench; Appendix D is the grading contract and the AI policy, and this is the week to reread it; Appendix E is the glossary.

Previously: Chapter 7 — trust in silicon: the root of trust, the chain derived from it, and the law that every optimization is a potential side channel.