Chapter 4 — Beyond the CPU

“General-purpose silicon is the tax you pay for not knowing what the workload will be.” — an industry adage

“and I have filled him with the Spirit of God, with ability and intelligence, with knowledge and all craftsmanship” — Exodus 31:3 (ESV)

Why This Matters

Open any modern mobile system-on-chip floorplan and ask how much of the die the CPU clusters occupy. The answer is uncomfortable for anyone who learned architecture as “the study of processors.” On a current phone SoC the CPU is a minority tenant, the GPU is often larger, and around them sit the neural processing unit, the digital signal processors, the image signal processor, the video codecs, the display pipeline and the modem — each built to do exactly one class of job, each there because a general-purpose core doing that job would cost more joules than the battery can afford.

That is this week. Chapter 1 taught you that mobile architects cannot buy watts. Chapter 2 showed the first response: make the cores unlike each other so the scheduler can pick a cheap one. Chapter 3 delivered the hard number underneath all of it — moving a word from DRAM costs orders of magnitude more energy than the arithmetic performed on it. Now the second response, and the more radical one: stop asking a general-purpose core to do the work at all. Build a machine whose dataflow is shaped like the problem, and the data stops moving.

Specialization is how you buy performance when you cannot buy watts. That is the thesis of Chapter 4, and everything in it — tile-based rendering, matrix engines, quantization, fixed-function video — is a variation on it. A fixed-function block does a narrow job at far better performance-per-watt than a general core because it does not pay for instruction fetch, decode, speculation, register renaming, coherence, or the possibility of being asked to do something else tomorrow. It pays for none of the generality, and generality is what costs joules. But specialization is not free, and that is the part a spec sheet never tells you: every fixed-function block is silicon that is dark for most workloads, and every accelerator narrows the set of programs the chip is good at.

Every accelerator also adds a decision — the fourth of The Four Questions, the one this chapter owns. Not how fast, not at what energy, not for how long before it throttles, but: which processor should do this work at all? That is placement, and this week you start answering it in writing, in a placement table you will carry to the capstone in Chapter 8.

AI enters here as the workload, not as a helper. On-device inference is the defining new mobile job and the reason the NPU exists — and it also teaches the most humbling lesson in the course: an accelerator with enormous nominal throughput can sit almost idle waiting on memory while your token rate crawls. The NPU is not a machine for doing more multiplies. It is a machine for not moving data, which Chapter 3 already told you is the same thing as a machine for saving energy.

The week’s question is the one the Bezalel passage in Exodus asks about a craftsman: who is fit for which work, and who decides? It is a question about vocation, and it is also, this week, a literal engineering question with a rubric attached. Hold both.

4.1 — Why Specialization Is the Only Move Left

A plugged-in machine that wants more performance has an old and reliable option: spend more power. Raise the clock, widen the core, add cores, add a fan. A phone cannot take that option — not because the silicon would fail, but because there is no thermal path to carry the heat away and no battery willing to fund it. Chapter 5 makes that precise with an energy budget and a skin-temperature limit. For now, take it as the boundary condition: the power budget is fixed, and it is small.

Inside a fixed power budget there is exactly one way to get more work done — make each unit of work cost fewer joules — and only so many ways to do that. Run at a lower voltage and frequency: real, bounded, and Chapter 5 owns it. Move less data: real, enormously effective, and Chapter 3 owns it. Stop paying for things you are not using: this is the one that produces accelerators.

Consider what a general-purpose out-of-order core spends energy on while executing a matrix multiply. It fetches and decodes instructions. It predicts branches, sometimes wrongly, and throws the work away. It renames registers and maintains a reorder buffer. It participates in a cache-coherence protocol. It checks permissions on every access. All of that machinery exists so the core can run any program you have not written yet — and every bit of it burns energy on a workload whose entire structure was known before the chip was taped out.

Now a matrix engine. No branch predictor, because there are no branches. No reorder buffer, because the dataflow is static. It reads a tile of weights once and reuses it across many activations, so the expensive DRAM traffic amortizes over hundreds of multiply-accumulates, and it works in eight-bit integers because the workload tolerates them. The multiply-accumulate is the same arithmetic the CPU would have done; everything the accelerator saves, it saves by not doing the surrounding work. Say that out loud until it is instinct: an accelerator is not faster arithmetic, it is less overhead and less data movement.

And here is the cost side, stated now so it colours everything that follows. Specialization narrows. A tile-based GPU is spectacular at the rendering pattern it was designed for and punishing when you violate it. A matrix engine that supports twenty operators is useless for the twenty-first. A video encoder that does H.264 and HEVC does not do the codec that ships next year. Every one of those blocks occupies die area — area that could have been cache, or another core, or nothing at all — and sits dark whenever its workload is not running. Generality has a value that does not appear on a spec sheet, and Chapter 8 ends the course on exactly that open question.

4.2 — The Processor Zoo: What Shape of Work Each One Wants

The single most useful mental model in this chapter is not a list of processors. It is a list of shapes of work, with a processor attached to each. Learn it this way round and placement becomes almost mechanical; learn it the other way round and you will spend your career guessing.

Two structural facts belong with that table, and both come straight out of Chapter 3. They all share the memory — CPU, GPU and NPU sit behind one memory controller and one DRAM interface, often with a system-level cache in front of it. That is a real win (no copy across a discrete-GPU boundary, because there is no boundary) and a real hazard: the GPU saturating memory bandwidth starves the CPU, and the NPU streaming weights competes with the display controller fetching the frame it must not miss. And they all share the power and thermal budget — two accelerators running at once do not each get the whole envelope. Which is why a placement decision that is correct for one workload in isolation can be wrong for six workloads at once, and why Chapter 5 will spend a week on it.

Coach’s Note — When someone hands you a workload and asks “CPU or GPU or NPU?”, do not answer. Ask three questions first: How wide is the parallelism? How regular is the control flow? How much data has to move per unit of arithmetic? Those three answers pick the processor nearly every time. The name of the workload — “it’s an AI thing”, “it’s a graphics thing” — picks it almost never.

4.3 — SIMD Versus SIMT, and the Cost of Divergence

Chapter 1 gave you SIMD: one instruction operating on a vector register, on one core, in one thread. NEON’s fixed 128-bit vectors; SVE2’s and RVV’s vector-length-agnostic model. That is the CPU’s answer to data parallelism, and it is bounded by the width of the register file and the number of cores you have.

A GPU answers differently. SIMT — single instruction, multiple threads — runs many independent threads, each with its own registers and its own program counter conceptually, but executes them in lockstep groups sharing one instruction stream. Vendors name the group differently: Nvidia calls it a warp, AMD a wavefront, Arm’s Mali a warp, Qualcomm’s Adreno a wave. The group width varies by vendor and by generation — commonly somewhere in the range of eight to a hundred and twenty-eight lanes — and you should never state a specific mobile width as fact without a datasheet in front of you. Chapter 8 will make you cite it.

The programming model is the important part: you write the code for one element and the hardware runs it for thousands. That is why GPU compute code looks like a scalar function and behaves like a vector machine, and why GPUs scale so gracefully across generations — widen the machine, run more groups, same source.

And then there is divergence, which is where the model bites. Inside one lockstep group all lanes share an instruction stream. When a branch sends some lanes one way and the rest the other, the hardware cannot execute both directions at once; it runs the first path with the non-taking lanes masked off, then the second with the others masked off. The two sides serialize, and the masked lanes burn time producing nothing.

// A divergent branch inside a lockstep group of N lanes.
// Half the lanes take the `if`, half take the `else`.
if (pixel_is_edge(x, y)) {
    heavy_edge_filter();     // lanes that fail the test idle here
} else {
    cheap_interior_path();   // lanes that took the `if` idle here
}
// Cost is NOT max(heavy, cheap). It is heavy + cheap, for every lane.

Do the arithmetic in the worst case. If a group of 32 lanes hits a branch where each lane takes a different one of 32 possible paths, the group runs at one thirty-second of its peak rate — while reporting, on any utilization dashboard, that the GPU is busy. Chapter 8 will teach you why “GPU utilization” is the most misleading number in mobile profiling; divergence is a large part of the reason.

Three practical rules follow. Make lanes in a group agree — sort or bucket work so neighbouring items take the same branch, because coherent work is fast work. Prefer arithmetic to branches when both sides are cheap — computing both and selecting often beats diverging. And move the genuinely irregular decision to the CPU, whose deep speculation is the right machine for it. That last one is a placement decision made for a structural reason, and it is exactly the kind of sentence your placement table needs.

4.4 — The Mobile GPU: Tile-Based Rendering, in Bytes

If you learn one mobile-specific architectural idea this term, learn this one. It is the clearest example in the course of an architecture designed around energy rather than peak throughput, and it explains more real mobile performance behaviour than anything else in the chapter.

The problem with the classic pipeline

A classic desktop immediate-mode renderer keeps the colour and depth buffers in external memory. Triangles arrive, are rasterized into fragments, and each fragment reads the depth value at its pixel, compares, and if it wins writes the new depth and the colour. All of that traffic goes to DRAM.

Count the bytes. At a phone resolution of 2400 × 1080 with four bytes of colour and four of depth per pixel, and an overdraw factor of 3 — three fragments shaded per final pixel, entirely ordinary in a real scene:

fragments        = 2400 × 1080 × 3                              =   7,776,000
bytes/fragment   = depth read 4 + depth write 4 + colour write 4 =          12
fragment traffic = 7,776,000 × 12                               =  93,312,000 B ≈ 89.0 MiB
final resolve    = 2400 × 1080 × 4                              =  10,368,000 B ≈  9.9 MiB
TOTAL to DRAM per frame                                          = 103,680,000 B ≈ 98.9 MiB

At 60 frames per second that is about 5.8 GiB per second of external traffic — for one application, on an interface the CPU, the NPU, the display controller and the camera also need. Chapter 3’s bandwidth arithmetic (data rate × bus width) puts a 64-bit-class LPDDR5X interface in the tens of gigabytes per second at best. You have just spent most of it drawing one frame, in the most expensive kind of byte on the chip.

The tile-based answer

Tile-based rendering restructures the pipeline around one observation: if the working set were small enough to keep on-chip, none of that traffic would happen. So make it small enough. Divide the frame into tiles — a tile of 32 × 32 pixels needs 32 × 32 × (4 + 4) = 8 KiB of storage for colour and depth, which fits comfortably in dedicated on-chip tile memory. Then:

1. Geometry pass (binning). Process all the frame’s geometry first. For each primitive, determine which tiles it touches and write its transformed data into a per-tile list — the bin or parameter buffer. This buffer goes to DRAM.
2. Rendering pass. For each tile in turn, read back only the primitives binned to it, then rasterize and shade them entirely inside tile memory with depth testing on-chip.
3. Resolve. When the tile is finished, write its colour to DRAM. Once. The depth buffer normally never leaves the chip at all — it was scratch space, and nobody needed it after the tile was done.

bin buffer write + read ≈ triangles × bytes-per-binned-primitive × (1 + tiles-touched)
colour resolve          = 2400 × 1080 × 4 = 10,368,000 B ≈ 9.9 MiB
depth traffic           = 0

The pixel-side traffic collapsed from ~89 MiB to ~9.9 MiB, and it no longer scales with overdraw at all — shading a pixel ten times over costs tile-memory bandwidth, which is cheap and on-chip, not DRAM bandwidth, which is expensive and shared. What replaced it is the bin traffic, which scales with geometry: at the model’s default of 150,000 primitives that is about 17 MiB, giving a tile-based total near 27 MiB against immediate mode’s 99. Same image, just over a quarter of the bytes.

The trade, made arithmetic

Tile-based rendering did not delete traffic. It exchanged per-pixel traffic for per-triangle traffic — a real trade with a real crossover, and code/tile_cost.cpp exists so you can find it rather than take my word for it:

g++ -std=c++17 -O2 -o tile_cost code/tile_cost.cpp && ./tile_cost

Sweep the overdraw factor at a fixed geometry load and the tile-based advantage grows steadily, because immediate-mode traffic scales with overdraw and tile-based traffic does not. Then hold overdraw at 1.0 — a scene with no overdraw at all — and sweep the triangle count upward instead, and you will watch the advantage shrink, hit parity, and invert. A geometry-dense, overdraw-free frame is where binning costs more than it saves. That is not a flaw in the architecture; it is the architecture being honest about what it optimized for. Know the assumption and you know when it fails.

TBDR: not shading what you cannot see

Tile-based deferred rendering goes one step further. Because all the geometry for a tile is known before shading begins, the hardware can resolve visibility first — determine which fragment wins at each pixel — and then shade only the winners. Hidden fragments are never shaded at all. Imagination’s PowerVR is the classic TBDR architecture and Apple’s GPUs are TBDR. Arm’s Mali and Immortalis GPUs are tile-based. Qualcomm’s Adreno can bin into tiles or render directly depending on the workload — Qualcomm calls that FlexRender — a reminder that tile-based versus immediate-mode is a spectrum and a runtime decision, not a permanent label.

Note what deferred shading saves and what it does not. It saves shading work — ALU cycles, texture fetches, and the energy of both. It does not change the DRAM traffic for the final colour resolve, which was already one write per pixel. Two different savings, two different mechanisms; do not conflate them in a report.

The rule a mobile graphics engineer says in their sleep

Do not read back a render target. Do not clear or load a render target you did not need. Do not sample the framebuffer you are currently drawing into. Every one of those forces the finished tile out to DRAM and often pulls the whole target back in again — the exact traffic the architecture was built to avoid. Run tile_cost and read the third output row: one gratuitous full-frame readback costs a write plus a read of the entire framebuffer and can erase most of the saving. It is one line of application code, and it is a bandwidth and energy catastrophe.

Coach’s Note — This is the section where the course’s thesis stops being abstract. Immediate-mode and tile-based rendering compute the same image. Same pixels, same result, same correctness. They differ only in how many bytes cross to DRAM — which is to say, they differ only in energy. A desktop architect can be indifferent between them. A mobile architect cannot. That is the whole discipline in one comparison, and it is worth having at your fingertips in an interview.

4.5 — GPU Compute: Borrowing the Graphics Machine

The GPU is a general data-parallel machine wearing a graphics costume, and mobile platforms give you several ways to use it directly.

The engineering content is the same in all of them, and it is not the syntax. It is this: a GPU dispatch is not free, and the memory is shared.

Every dispatch costs something — setting up the pipeline, making the data visible to the GPU, synchronizing at the end — and for a small work item that overhead can exceed the work. Look at the wake_word rows in code/placement-bench.csv: in that synthetic dataset the GPU is faster than the CPU and costs more than twice the energy, because powering up a wide parallel machine for a tiny model is the wrong trade in every direction.

And unified memory does not mean free memory. It means no copy — a real win over a discrete GPU across a PCIe-class boundary — but also one memory controller, one DRAM interface, and one bandwidth budget shared with every other block. A compute kernel that streams a large buffer competes directly with the display controller, whose deadline is not negotiable. Chapter 6 shows what the interconnect does about that.

Between them, those two facts explain the most common disappointment in GPU compute: a kernel that is genuinely faster in isolation and slower in the application. The cause is almost never the kernel. It is dispatch overhead on work that was too small, or bandwidth contention with something else on the SoC. Measure the application, not the kernel — Chapter 8 will make you say which one you measured, every time.

4.6 — The NPU: A Machine for Not Moving Data

Here is where Chapter 3’s energy lesson pays off, restated precisely and with its citation. In his ISSCC 2014 keynote “Computing’s Energy Problem (and what we can do about it)”, Mark Horowitz laid out the energy cost of operations at a 45 nm process node. The widely quoted figures: a 32-bit integer add on the order of 0.1 pJ; a 32-bit read from an 8 KB SRAM on the order of 5 pJ; a 32-bit DRAM access on the order of 1300–2600 pJ. Absolute numbers move with process node and design; the ratio does not move much, and the ratio is the point. Fetching an operand from DRAM can cost three to four orders of magnitude more energy than the arithmetic you then perform on it.

Now look at what a neural network’s inner loop is made of. Load a weight. Load an activation. Multiply. Accumulate. Repeat, billions of times. If every weight and activation makes a round trip to DRAM, the arithmetic is a rounding error in the energy bill and the memory system is the entire cost.

So an NPU is designed backwards from that fact. Its defining feature is not the multiplier array — CPUs and GPUs have multipliers too. It is a dataflow that maximizes reuse on chip: a MAC array arranged so a value loaded once is consumed by many units (which operand is held stationary — weight, output accumulator, or activation row — is the design choice the literature calls dataflow); local buffers sized so a tile of weights and a tile of activations stay resident while a whole block of output is computed against them; low precision by default, INT8 and INT4 rather than FP32, because a narrower operand is a smaller multiplier, less buffer, and above all fewer bytes to move; and a static schedule, because the graph is known in advance, so there is no branch prediction and no reorder buffer.

The names to use correctly, as of 2026: Apple’s Neural Engine; Qualcomm’s Hexagon, which serves as both DSP and NPU; MediaTek’s APU; the TPU-derived machine-learning block in Google’s Tensor SoCs; Samsung Exynos’s NPU; and Arm’s Ethos NPU IP, licensed into many designs. Learn the names. Do not learn the marketing numbers attached to them — see §4.13 and Chapter 8 on why TOPS tells you almost nothing.

Coach’s Note — Here is the test for whether you have understood this section. Someone tells you a new accelerator has twice the multiply-accumulate throughput of the previous one. What is your first question? It is not “how fast is it.” It is “did the memory path get wider too?” — because if the weights still arrive at the same rate, the extra MACs will be idle and you will have paid area and leakage for nothing. This is the accelerator version of the memory wall, and it is why the NPU section of this chapter is a memory section wearing a compute hat.

4.7 — Quantization: The Cheapest Four-Times You Will Ever Get

Quantization is storing and computing on weights and activations in fewer bits: FP32 → FP16 or BF16 → INT8 → INT4. It is the single most consequential software decision in on-device inference, and part of it is exact arithmetic while part of it is irreducibly empirical. Keeping those two parts separate is most of the skill.

The exact part. An FP32 weight occupies 4 bytes; an INT8 weight occupies 1. That is a 4× reduction in weight memory, exactly — not an estimate, not a benchmark, arithmetic. INT4 is 8× against FP32. Since Chapter 3 established that on-device inference is usually bound by memory rather than by arithmetic, a 4× cut in bytes moved is very close to a 4× cut in the thing that actually costs you. It is also a 4× cut in the footprint that decides whether the model fits on the device at all.

The mechanism. Symmetric integer quantization is three operations. Pick a scale: scale = max(|w|) / 127. Quantize: q = clamp(round(w / scale), -127, 127). Dequantize when you need the real value back: w' ≈ q × scale. That is it. code/quantize.py implements exactly those three lines in plain Python with no dependencies, so you can satisfy yourself that nothing is hidden.

The choice that matters: per-tensor or per-channel. A per-tensor scale uses one scale for the whole weight matrix; a per-channel scale gives each output channel its own. That sounds like bookkeeping and is not. Run python3 code/quantize.py. The first matrix is well-behaved and the two schemes are nearly identical. The second is deliberately hostile: channel 0 carries values roughly forty times larger than the rest. Watch the mean err, quiet ch. column — the error on the other channels, the ones that did nothing wrong. Under a per-tensor scale it explodes, because one global scale had to stretch to cover the loud channel and every quiet channel is now represented in a handful of the available levels. Under per-channel scales it barely moves.

One loud channel poisons every quiet channel it shares a scale with. That is the entire argument for per-channel quantization, and now you have watched it happen rather than been told.

PTQ versus QAT. Post-training quantization takes a trained FP32 model and quantizes it, usually with a small calibration set to choose scales. It is fast, needs no training pipeline, and is where you start. Quantization-aware training simulates quantization during training so the network learns weights that survive it; it costs a training run and recovers accuracy PTQ loses. The choice is empirical: try PTQ, measure, escalate only if the measurement says you must.

And the rule that outranks everything else here. Accuracy loss under quantization is workload-dependent and must be measured, never assumed. Not estimated from a blog post. Not inferred from “INT8 is usually fine.” Not taken from a language model that will produce a confident, specific, entirely invented accuracy-drop percentage if you ask it. Measure it on your model, on your task, with your evaluation set, and report the number with the conditions under which you got it.

Coach’s Note — Watch for the paper that reports a quantized model’s average accuracy holding steady while a specific, important sub-case collapses. Aggregate metrics hide exactly the failures a product cannot ship with. If your model does one thing that matters much more than the rest — recognizing a medication name, refusing an unsafe instruction, reading a specific field on a form — measure that thing separately before and after. Chapter 8 will call this the difference between a true number and an honest one.

4.8 — On-Device Inference Is a Memory Problem

Now put quantization and the NPU together and do the arithmetic that decides whether an on-device model is viable. All of this is arithmetic on stated assumptions; replace every assumption with your own model’s real numbers before you cite it.

Weights. weight bytes = parameters × bytes-per-weight. A 3-billion-parameter model:

On a phone whose entire DRAM is shared with the OS, the display, the camera and every running app, the difference between the top row and the bottom row is the difference between “impossible” and “shipping.”

The KV cache. For a transformer generating tokens autoregressively, the key/value cache grows with sequence length and is the footprint people forget:

KV bytes = 2 (K and V) × layers × sequence_length × kv_heads × head_dim × bytes_per_element

Take a hypothetical configuration — 28 layers, 8 key/value heads, head dimension 128, FP16 elements, 4096 tokens of context — purely to show the shape of the arithmetic:

2 × 28 × 4096 × 8 × 128 × 2 bytes = 469,762,048 bytes = 448 MiB

Nearly half a gigabyte, on top of the weights, growing linearly with context length: double the context, double the cache. On-device context windows are a hardware decision as much as a product one, and this is what the memory-footprint dataset you met in Chapter 3 (model-memory.csv) was setting up.

Why it is memory-bound, in one calculation. Generating one token in a dense model requires reading essentially every weight once and performing roughly two operations per weight — one multiply, one add. At batch size one:

bytes moved per token (INT4, 3B params) = 3e9 × 0.5 = 1.5 GB
operations per token                    = 2 × 3e9   = 6e9
arithmetic intensity                    = 6e9 ops / 1.5e9 bytes = 4 operations per byte

Four operations per byte is a catastrophically low arithmetic intensity. Now assume — and label it an assumption, not a claim — an achievable 50 GB/s, comfortably under the roughly 68 GB/s peak that Chapter 3’s arithmetic gives for a 64-bit LPDDR5X interface:

time per token ≥ 1.5 GB ÷ 50 GB/s = 30 ms   →  about 33 tokens per second

That floor is set entirely by memory. The MAC array could be ten times wider and the number would not move, because the weights cannot arrive any faster. On-device token generation is a bandwidth problem wearing a compute costume. Every architectural response follows: quantize to move fewer bytes; use grouped-query attention to shrink the KV cache; keep what you can resident on-chip; batch when you legitimately can, which on a single-user device is almost never.

Coach’s Note — This is the calculation to do before you promise a product manager a token rate. Weights ÷ achievable bandwidth is a hard floor, computable in thirty seconds, and it is wrong only in the pessimistic direction. If the floor already misses your target, no amount of accelerator selection will save you, and the honest answer is a smaller model, a lower precision, or a different feature.

4.9 — Placement: Partitioning a Workload Across the Fleet

You now have the pieces. Here is the discipline that assembles them, and the artifact this chapter contributes to the rest of the course.

The placement table has four columns: workload → chosen processor → why → what it costs. It is not a spreadsheet of benchmark scores. Every row is an argument, and a row without a defensible “why” and an honest “what it costs” is a preference, not a decision. You produce your first one this week, and Chapter 8 requires it in the capstone.

The tool for building it is code/partition.py, which reads the six-workload dataset in code/placement-bench.csv, applies a latency budget you state, and returns the lowest-energy assignment that meets it — with everything it rejected and why.

python3 code/partition.py --budget-ms 50
python3 code/partition.py --budget-ms 20 --verbose

That dataset is synthetic. Its first line says so, and you must say so too in anything you submit that uses it. Its ratios are modelled on published architectural behaviour; its absolute numbers measure nothing and may not be cited as though they did. Appendix B documents its provenance along with every other dataset in the book.

Run it at both budgets and watch llm_token_gen. At 50 ms it lands on the NPU: slower than the GPU, less than half the energy. Tighten to 20 ms and the NPU’s 24 ms no longer fits, so the assignment flips to the GPU — which meets the deadline and costs roughly twice the joules. Nothing about the hardware changed. The requirement changed, and the correct answer changed with it. That is what it means to say placement is a decision rather than a fact, and it is why a placement table without its stated budget is meaningless.

Operator fallback: the classic disappointment

Here is the failure mode every practitioner has hit and no tutorial warns about. You hand a model graph to an accelerator runtime. The runtime walks the graph and finds one operator it does not implement — a custom activation, an unusual reshape, a dynamic-shape node. It does not fail. It partitions: it runs what it can on the accelerator and falls back to the CPU for the rest, so the tensors now round-trip — accelerator to memory, memory to CPU, CPU back to memory, memory back to the accelerator — possibly several times per inference.

The result is an “accelerated” path that is slower than pure CPU, and often hungrier, while every dashboard reports that the NPU is engaged. Look at the photo_pipeline rows in the dataset: the NPU row is worse than the CPU row on both latency and energy, and its note names the cause. partition.py flags it whether or not it wins, because you need to see the trap before you defend your assignment, not after.

The professional response is short and unglamorous: profile the partition. Ask the runtime which operators it actually placed on the accelerator, and compare the accelerated path against a plain CPU baseline on the same input, every time. Assume nothing about coverage.

The Android acceleration landscape, as of 2026

State this carefully, because it is moving. NNAPI is deprecated — deprecated in Android 15, API level 35. For most of the last decade it was the answer to “how does an Android app reach the NPU,” and it is no longer that answer. The current direction is framework-level delegation: TensorFlow Lite / LiteRT with vendor delegates (Qualcomm’s QNN, MediaTek’s NeuroPilot), vendor SDKs used directly, GPU delegates for the graphics path, and on-device machine-learning services delivered through Google Play services. Apple platforms route through Core ML, which decides for itself among CPU, GPU and Neural Engine.

The honest summary is that the acceleration path is vendor- and framework-specific, and still moving. That is not a gap in your knowledge; it is the actual state of the field, and the professional consequence is that you check current documentation for the platform you are shipping on rather than trusting a two-year-old tutorial — or a language model whose training data froze somewhere in the middle of the transition.

Coach’s Note — This is the week’s AI-as-untrustworthy-assistant lesson, and it is unusually sharp here because the ground moved recently. Ask an assistant “how do I run a model on the Android NPU” and there is a real chance you get a confident, well-formatted, obsolete answer built around NNAPI, complete with plausible-looking code. Use the model to explain what a delegate is and why fallback happens — it is genuinely good at that. Never use it to source what the current API is. Appendix D states the policy in one line: use it to explain, never to source. An uncited figure in a submitted report is a rubric failure, not a deduction.

4.10 — The Fixed-Function Fleet: ISP and Video

Two blocks round out the chapter, and both make the specialization argument in its purest form.

The image signal processor is a fixed-function pipeline between the camera sensor and everything else: demosaicing (turning the sensor’s colour-filter mosaic into full-colour pixels), noise reduction, lens shading and distortion correction, tone mapping, and — the modern part — multi-frame fusion, combining a burst of exposures into one image with better dynamic range and less noise than any single frame. This is why a phone camera outperforms its optics: the optics are small and the computation is enormous. Increasingly the ISP and the NPU cooperate on the same capture, with learned models handling stages the fixed-function pipeline handles poorly — which is exactly what the photo_pipeline rows in the dataset are a miniature of.

Video encode and decode blocks are fixed-function for the reason this chapter has been building toward. Running H.264, HEVC or AV1 in software burns orders of magnitude more energy than the dedicated block, and on a phone that is not a benchmark result, it is whether the battery survives a video call. AV1 decode is broadly present in recent flagship mobile silicon; AV1 encode in hardware is less widespread, so check the specific part rather than assume.

Now look at the video_encode_4k30_1s rows in code/placement-bench.csv and notice what is missing. There is a CPU row (an energy disaster), a GPU row, a DSP row, and an NPU row marked UNSUPPORTED because entropy coding is serial and bit-exact and is simply not a tensor graph. There is no fixed-function video encoder row at all — and the fixed-function encoder is the correct answer.

That omission is deliberate and it is graded. The most valuable conclusion a placement analysis can reach is sometimes “none of the processors on this list, and here is the block you actually need.” An architect who can only choose among the options presented is not doing architecture. Chapter 6 shows what it costs to wire such a block into the system; Chapter 8 asks what it costs to leave it dark when nobody is recording video.

4.11 — Interactive Lab: The Workload Placement Bench

Below this chapter on the website you will find The Workload Placement Bench — the centrepiece interactive of the course, and the one that turns §4.9 from a paragraph into a reflex. Use it now; it is part of the chapter, not an extra.

The bench gives you six workloads — image classification, 4K video encode, a game frame, a speech wake-word, a small language model’s token generation, and a photo pipeline — and lets you assign each to a processor, returning latency, energy, a thermal contribution and a verdict explaining why that pairing behaves the way it does. Work through it deliberately:

1. Predict before you click. For each workload write down the processor you expect to win and the reason, using §4.2’s three questions: how wide is the parallelism, how regular is the control flow, how much data moves per unit of arithmetic. Then assign it and read the verdict. The gap between your prediction and the panel is the lesson; the panel agreeing with you teaches you nothing.
2. Walk into the operator-fallback trap on purpose. One of the six has an accelerator path that looks obviously correct and is not, because part of its graph falls back to the CPU. Assign it the intuitive way, read what the round trips cost, then decide where it really belongs. Meet this here, where it costs you nothing.
3. Switch to budget mode. You get a latency ceiling and an energy ceiling and must assign all six at once; the bench grades the whole assignment. Harder than it looks, because the cheapest choice for one workload can push another over its deadline — and because at least one workload’s honest answer is that no listed processor should run it.

What it teaches: placement is a constrained optimization with stated requirements, not a ranking of processors. The same silicon yields a different correct answer under a different budget.

4.12 — Filled with Skill for the Work

In Exodus, before a single article of the tabernacle is built, God names a craftsman. Bezalel is filled “with the Spirit of God, with ability and intelligence, with knowledge and all craftsmanship” (Exodus 31:3, ESV). It is a strange verse to find in the middle of construction specifications, and the strangeness is the point: the text stops to say that fitness for the work was given, particular, and named. Not general excellence. Skill for this craft.

That is a remarkably good description of an SoC floorplan. Each block on it is a worker gifted for one task: the ISP cannot generate a token, the matrix engine cannot rasterize a triangle, the video encoder cannot decide anything. And the arrangement is not a hierarchy — the CPU is not the important one delegating to lesser assistants. The CPU is also a specialist, in irregular control flow and low-latency decisions, and it is precisely as bad at dense linear algebra as the NPU is at branching. Chapter 2 made this point about cores and the body’s members; Chapter 4 makes it about whole processors, and the temptation to rank is stronger here because one of the blocks runs your operating system.

So the week’s question — who is fit for which work, and who decides? — has a technical answer and a vocational one, and they illuminate each other. The technical answer is placement, and its discipline is humility about your intuitions. You do not know which processor is fit for a workload from what the workload is called. “It’s an AI workload, so it goes on the NPU” is the operator-fallback trap wearing a name badge. You know by examining the shape of the work and then measuring. The architect’s authority comes from having looked.

The vocational answer runs alongside it. There is real dignity in the specialized worker, and the passage grants it without apology: the craftsman filled with skill for the tabernacle is not a lesser figure than the prophet. In the LCMS tradition this is the doctrine of vocation — ordinary particular work, done well in the station where you have been placed, is genuinely God-pleasing work and not a consolation prize for people who could not do something more visible. The engineer who spends a career on image pipelines has a calling, not a niche.

But hold the hard edge too, because a chapter that only flatters specialization has not told you the truth. Every fixed-function block is silicon that is dark for most workloads. Every gift is also a limit. An architecture that specializes without restraint ends up with a chip full of workers who have nothing to do most of the time — area spent, leakage paid, flexibility surrendered against a future workload nobody has described yet — and you discover the cost of having abandoned generality only when the workload changes. Knowing when to specialize and when to keep the general-purpose path is not a calculation; it requires honesty about what you do not know about the future, which is the posture the Bezalel passage assumes. Gifted for the work in front of you, inside a design you did not author, not confusing your competence with omniscience. Chapter 8 ends the course on that question and does not resolve it, because it is not resolved.

4.13 — Common Pitfalls

Pitfall: Assuming that anything called “AI” belongs on the NPU. Example: A photo pipeline is assigned to the NPU because it contains a learned model; two custom operators are unsupported, the graph is split, and the tensors round-trip to the CPU three times per capture. The “accelerated” path is slower and hungrier than plain CPU. Fix: Profile the partition. Ask the runtime which operators it actually placed on the accelerator, and always compare against a plain CPU baseline on the same input. partition.py flags the fallback row in the dataset whether or not it wins — that flag is the habit, not the tool.

Pitfall: Reading back or gratuitously reloading a render target on a tile-based GPU. Example: A post-processing effect samples the framebuffer it just wrote. One API call forces the finished tile out to DRAM and pulls the whole target back in. Fix: Restructure so the effect happens inside the same render pass, or use a subpass/tile-local mechanism where the API offers one. Run code/tile_cost.cpp and read the readback row: it can erase most of the tiling advantage in a single call.

Pitfall: Comparing accelerators by TOPS. Example: Two parts are ranked by their quoted “AI performance” figures and the larger number is declared the better machine. Fix: Ask the three questions TOPS omits: at what precision, at what utilization, and sustained for how long? A peak integer-operation rate says nothing about achievable throughput on your graph, nothing about memory bandwidth — which §4.8 showed is usually the binding constraint — and nothing about what survives thermal steady state. Chapter 8 treats this as a measurement-honesty problem, which is what it is.

Pitfall: Quantizing to INT8 and shipping on the strength of “INT8 is usually fine.” Example: Aggregate accuracy holds within a point, so the model ships — and the one sub-case the product exists to handle degrades badly, discovered by users. Fix: Measure accuracy on your task with your evaluation set, before and after, and measure the important sub-case separately from the average. If PTQ loses too much, escalate to quantization-aware training — but escalate on a measurement, not a hunch. Run code/quantize.py to see how completely the error depends on the data.

Pitfall: Quoting an accelerator’s peak number as though it were what the device delivers. Example: A placement table justifies an NPU assignment with a peak throughput figure measured in a thirty-second burst on a cold device. Fix: Peak and sustained are two different claims and you must say which one you are making. A sustained accelerator workload shares one thermal envelope with everything else on the SoC, and the steady state is not the burst. Chapter 5 is entirely about why, and it will make you measure the difference yourself. Chapter 4’s placement numbers are all peak-ish; Chapter 5 is where they get re-examined.

Pitfall: Choosing only among the processors on the list you were handed. Example: A 4K video encode is assigned to the GPU because it is the best of the four options in the dataset, and the analysis stops there. Fix: Ask whether the right answer is a block that is not on the list. The honest conclusion “none of these; this workload needs a fixed-function encoder, and here is what that costs in area and in dark silicon” is a stronger piece of architecture than the best available compromise — and it is the one this week’s practical grades you on.

4.14 — Reps

Open the exercises and work all of it. This week’s reps build the two reflexes the practical grades: pricing an architectural choice in bytes and joules rather than in adjectives, and defending a placement against the alternatives you rejected.

This week’s AI policy. Use an assistant to explain — what a lockstep group is, why per-channel scales exist, what a delegate does. Do not use it to source. Every figure that lands in a submitted artifact needs a primary source: a vendor specification page, an architecture reference manual, this book’s fact-grounded text, or your own run of the shipped code. A confidently fabricated cache size, group width or TOPS rating in a report is a rubric failure, not a deduction. Appendix D states the policy; Appendix A tells you which workbench you are on.

A preview:

• Rep 2 — Price a divergent branch inside a SIMT group, and say what the idle lanes were doing.
• Rep 4 — Run code/tile_cost.cpp and read the overdraw sweep from immediate mode to tile-based.
• Rep 7 — Hand-compute the INT8 memory saving, then run code/quantize.py and check yourself.
• Rep 9 — Build the placement table under a stated latency budget and defend every row you kept.
• Rep 10 — Walk into the operator-fallback trap on purpose and measure what the round trips cost.

A short “Check Your Reps” quiz sits on this page below the lab. Like the on-page quiz in every chapter, it is an ungraded self-check — take it anyway, because this week it is the closest thing you will get to a rehearsal for the checkpoint. Week 4’s graded knowledge check is not an ordinary weekly quiz; see §4.15.

4.15 — This Week’s Project

There is no ordinary weekly lab this week. Week 4 is the checkpoint, and it carries two graded items instead — one that tests what you know, and one that tests what you can decide.

The graded knowledge check is the cumulative auto-graded checkpoint, worth 10% of your course grade. It is not a Chapter 4 quiz. It draws cumulatively across Weeks 1 through 4 — instruction sets and exception levels, heterogeneous cores and migration cost, the memory hierarchy and the energy of data movement, and this week’s accelerators. Everything since Week 1 is in scope. The Week 4 practice quiz is ungraded; it exists so you can rehearse without stakes, and skipping it is a choice you will feel on the checkpoint.

Your deliverable is the Week 4 Placement Practical — a take-home worth 8% of your course grade, issued in Canvas. You will run code/quantize.py and code/tile_cost.cpp for real on your own machine, drive code/partition.py over code/placement-bench.csv under a stated latency budget, and produce two documents: placement-decision.docx, the placement table for all six workloads with the energy arithmetic behind every row, and report.docx, defending it. The graded core is not the table. It is the defence — why each row beat the alternatives, what the operator fallback does to a naive assignment, which workload’s correct answer is a block that is not in the dataset at all, and, the line that separates a graduate answer from an undergraduate one, an explicit statement of what you would need to measure on real hardware to confirm any of it. The dataset is synthetic; say so plainly, and say what it limits.

Medium and Hard tiers add extra credit as in every other week. Medium re-runs the whole placement under a different budget and reports which rows flip and why; Hard is the architecture memo, where you decide whether to spend die area on one more block and defend it in joules against the cost of silicon that will be dark most of the time. That last one is judgment, and judgment is what this course grades.

Stated once this week, consistent with Appendix D: six weekly labs at 7% each (42%), this week’s practical (8%), the Week 4 checkpoint (10%), eight weekly knowledge checks at 1.5% each (12%), the capstone investigation (20%), and the recorded technical briefing (8%). Note how Week 4 stacks: the checkpoint is this week’s knowledge check rather than an extra item beside it, so it carries the 10% and the 1.5% together — 11.5%, the second-largest single item in the course.

4.16 — Coach’s Final Word

You came into this week able to describe a phone’s processors. You leave it able to choose one, and to say in bytes and joules why the others lost. That is a different skill, and it is the one the industry is short of.

Look at what you can now do. You can price tile-based rendering against immediate mode in megabytes per frame, and you know why reading back a render target is an energy catastrophe rather than a style violation. You can compute a model’s weight footprint and its KV cache exactly, divide by an achievable bandwidth, and produce a token-rate floor before anyone writes a line of code. You know the 4× from INT8 is arithmetic and the accuracy loss is an experiment, and you have watched one loud channel poison a whole tensor. And you know the shape of the operator-fallback trap well enough to check for it before you defend a placement.

Hold on to the fourth question. Performance, energy, thermals, placement — this chapter owns the fourth, and it is the one that turns architecture from a description into a decision. A placement table is not a list of components; it is a set of arguments, each with a stated budget, a rejected alternative, and an honest account of what the choice costs. Write it that way this week and you will write it that way in the capstone.

Bezalel was filled with skill for a particular work. So is every block on that floorplan, and so, in a smaller and more ordinary way, are you — gifted for the work in front of you, inside constraints you did not choose, with no obligation to pretend you can see further than you can. Do the arithmetic. Name what you could not measure. Then decide anyway, and sign your name to it.

Next week the floor moves. Everything you decided this week gets re-examined under a power and thermal budget that does not care how elegant your placement was, and you will find out which of your answers were true only for the first ninety seconds.

See you next week.

Up next: Work every rep in the exercises — they are the conditioning, and they rehearse the practical in miniature. Then take the Week 4 practice quiz (ungraded), sit the cumulative checkpoint covering Weeks 1–4, and complete the Week 4 Placement Practical in Canvas. Toolchain and workbench questions go to Appendix A; the datasets and their provenance are documented in Appendix B; how to report a measurement honestly is Appendix C; grading is Appendix D; the vocabulary is Appendix E. Then open Chapter 5 — the budget that governs everything, and the week that re-reads Chapters 1 through 4 through the energy lens.

Previously: Chapter 3 — the real bottleneck: caches, locality, LPDDR bandwidth, and why moving a word costs more than the arithmetic on it.