Every lecture in Part II has attacked the numerator of Lecture 2's decode floor: fewer reserved bytes, fewer wasted slots, fewer cached bytes, fewer bits per weight. Today attacks the denominator. The floor says one pass over 13.5 GB of weights yields one token, and there is no law requiring that — a cheap draft model can propose several tokens and the expensive model can check all of them in a single pass, because checking k tokens is a prefill-shaped operation and prefill is compute-bound. What makes this deployable rather than a heuristic is that the verification is exactly distribution-preserving: a short rejection-sampling argument shows the output is drawn from the same distribution as ordinary decoding, so unlike everything in the last two lectures it needs no quality evaluation at all. What makes it interesting is that it is fundamentally a trade of spare FLOPs for latency — so it works best where the machine is emptiest, and at a large enough batch it becomes a net loss. By the end you should be able to derive the expected accepted length from an acceptance rate, choose the optimal draft length, and compute the batch size at which speculation stops paying.

Where this sits

Lecture 2 §2.13 established the decode floor: at batch 1 a step must stream all 13.5 GB of weights to produce one token, which is 4.0 ms on an H100 and uses 0.34% of its arithmetic. Every serving lecture since has attacked the bytes. Lecture 10 stopped reserving them, Oct 5 stopped wasting slots, Oct 19 shrank the cache by up to 25.6×, and Nov 9 shrank the weights themselves — and after all of that, Lecture 14 §14.7 found the step was 59% weights and handed the problem forward.

Today is the other half of that hand-off, and it is the last lecture on single-request efficiency in the course. It takes the ratio tokens per pass over the weights and raises it above one for the first time. That is also why it belongs immediately before the agent-serving block: Nov 18 and Nov 23 stop optimizing the individual request and start optimizing the program that issues hundreds of them, and speculation is the natural bridge, because an agent's highly templated prompts and structured outputs are the most speculation-friendly traffic there is.

Instructor notes — Discussion plan

75-minute class, student-led. Two required papers with a clean division of labour — one owns the correctness proof, one owns the acceptance rate — and one instructor-held result (§16.8) that contradicts how the technique is usually described.

Time	Segment	Who	Notes
0–7	The denominator, and the spare FLOPs	instructor	Re-board 0.34%. "99.66% of the arithmetic is idle. Spend it."
7–22	Presenter 1: the original	student	The rejection-sampling rule. Force the "why is it exact?" proof at the board.
22–34	§16.4 The arithmetic	instructor	Protected — derive live. E[accepted], then the α×k table and the moving optimum.
34–48	Presenter 2: EAGLE	student	Feature-space drafting. The acceptance table is the result.
48–56	§16.7 Trees	room	Why a tree beats a chain at equal verification cost.
56–70	§16.8 When it stops helping	instructor	Protected. 2.10× → 1.33× at B = 164, loss above B ≈ 324. Then MagicDec's inversion.
70–75	Wrap	instructor	The one-line law, then point at Nov 18.

Questions to force. (1) "Is the output identical, or identically distributed? Does the difference matter?" (2) "Why is verifying k tokens cheap?" — push to prefill-shaped and compute-bound. (3) "Your engine runs at B = 164 to hit its throughput target. Do you enable speculation?" (4) "What property of agent traffic would make α unusually high, and how would you measure it rather than assume it?"

Reading-only, not scheduled: §16.6's EAGLE-2 material and §16.7's Sequoia tree-optimization discussion.

If running long: compress §16.7 to the one-sentence tree argument. Never cut §16.4 or §16.8 — the acceptance arithmetic is what makes the reading list legible, and §16.8 is the only place students will be told that the technique has an operating range.

Learning objectives

By the end of this class you should be able to:

1. Explain why one forward pass yielding one token is an accident of the algorithm rather than a hardware constraint, and quantify the spare arithmetic that makes speculation possible.
2. State the draft-and-verify loop, and explain why verifying k tokens costs about the same as generating one.
3. Reproduce the rejection-sampling acceptance rule and argue that the resulting output distribution is exactly the target's.
4. Derive the expected number of accepted tokens per verification from α and k, and the resulting speedup including the draft's cost.
5. Choose the optimal draft length for a given α and draft cost, and explain why the optimum grows with α.
6. Explain what EAGLE changes and why predicting a feature is easier than predicting a token.
7. Explain why a tree of drafts dominates a chain at equal verification cost.
8. Compute the batch size at which speculation becomes a net loss, and explain why long context reverses the conclusion.

16.1 The denominator nobody had attacked

Return to the floor, and read it as a ratio rather than as a duration.

A batch-1 decode step moves 13.5 GB and performs 13.48 GFLOP — one FLOP per byte, 1/295 of the machine's arithmetic, 0.34%. Stated as a bill: the step's cost is set entirely by the weight streaming, and having paid it, the engine extracts exactly one token. Every other lecture in Part II tried to make the payment smaller. Nobody asked for more goods.

There is no reason the second term must be one. If the pass could be made to yield n tokens, the per-token cost would fall by n with the bytes unchanged — and the arithmetic to do it is already sitting idle, because at 0.34% utilization the tensor cores are doing essentially nothing for the whole 4 ms. Speculative decoding is the technique that spends idle FLOPs to buy latency, and keeping that framing in view explains every result in the lecture, including the unwelcome one in §16.8: a technique that spends spare capacity stops working when the capacity stops being spare.

The obstacle is causality. Token i+1 depends on token i, so you cannot compute several tokens in one pass — unless you already know what they are, in which case you are not computing them but checking them. And checking is cheap for exactly the reason Lecture 9 §9.2 and Lecture 12 §12.1 established: k tokens through one pass over the weights is a prefill-shaped operation, compute-bound and far past the ridge point, so its cost is dominated by the same weight read a single-token step already pays.

That box is the whole idea. The rest of the lecture is (a) how to produce plausible candidates, (b) how to check them without changing the output distribution, and (c) when the "still 4.05 ms" stops being true.

Instructor notes

Minutes: 7. Instructor holds this; no paper yet. Board: "1 pass → 1 token" with "0.34%" beside it. Then cross out the 1 and write n. Ask the room: "Where would the extra tokens come from?" Let them arrive at "guess, then check". Expect confusion: Students think verification must be sequential too. Draw the five positions going through one pass and point at the single weight read.

16.2 Draft, verify, repeat

The loop is four lines and worth stating precisely because the papers' variations are all variations on it.

A cheap draft model proposes k candidate tokens autoregressively — so k cheap sequential passes. The target model then runs one pass over all k+1 positions (the k proposals plus the one it would generate itself), obtaining its own distribution at every position. The acceptance rule (§16.3) walks the proposals left to right, accepting each while it passes a test, and stops at the first rejection. On rejection, the target's own corrected token is emitted at that position, so every verification pass yields at least one token — the algorithm can never go backwards. Then repeat from the new context.

Three properties of that loop matter for a serving system.

Progress is guaranteed. Even with a useless draft the loop degenerates to ordinary decoding plus the wasted draft cost, so the downside is bounded by k·c rather than by unbounded stalling.

The draft runs k sequential passes, so its cost is k·c target-equivalents, and a draft that is one tenth the size but must run four times is not free. This is why c appears in every formula and why the draft's depth is as important as its size.

The draft needs the same context. It must attend over the same prefix, so it keeps its own KV cache — a second cache, at its own (smaller) per-token cost. For a 1B draft against our 7B target that is roughly 1B/6.74B of the KV per token at comparable architecture, so single-digit percent of the budget; it is not free, and at high batch it competes with the target's cache for the same 62.5 GB.

Instructor notes

Minutes: included in presenter 1's slot. Board (presenter): Four boxes — draft ×k, verify ×1, accept-prefix, emit — in a cycle, with "≥1 token per cycle" written under it. Force the question: "What happens if the draft is terrible?" Degenerates gracefully; that is a design virtue worth naming.

16.3 Why the output distribution is exactly preserved

This is the section that makes the technique deployable, and it is the reason speculation sits on the opposite side of Lecture 14 §14.8's line from everything in that lecture. Let p be the target's distribution at a position and q the draft's, and let x be the draft's proposal.

The claim is that the emitted token is distributed exactly as p. The argument is short enough to do at the board. For any token x, it can be emitted two ways. Accepted: probability q(x) · min(1, p(x)/q(x)) = min(q(x), p(x)). Or emitted after a rejection: the probability of rejecting is 1 − Σ_y min(q(y), p(y)), and the residual distribution assigns x a share max(0, p(x) − q(x)) ⁄ Σ_y max(0, p(y) − q(y)). Since Σ_y max(0, p − q) = 1 − Σ_y min(p, q), the rejection branch contributes exactly max(0, p(x) − q(x)). Adding the two branches:

min(p(x), q(x)) + max(0, p(x) − q(x)) = p(x)

identically, for every x. No approximation, no tuning parameter, no dependence on how good q is.

Three consequences worth stating, because they are what distinguish this from every other optimization in the last three lectures.

It needs no quality evaluation. Lecture 14 §14.8 demanded a frozen task set and an interval for any technique that could change an output. Speculation cannot change the output distribution, so it owes a performance measurement only. That is a genuine engineering advantage and it is why speculation ships enabled by default in engines that keep cache quantization behind a flag.

"Identically distributed" is not "identical". A given run with speculation will generally produce a different sample than a given run without it, because the random draws differ — the same way two runs with different seeds differ. What is preserved is the distribution, which is the correct thing to preserve and is exactly what temperature-based sampling already assumes. At greedy decoding (temperature 0) the guarantee becomes token-for-token identity, since p is a point mass and the rule accepts precisely when the draft agrees with the argmax.

The draft's quality affects speed only. q appears nowhere in the output distribution, so a bad draft costs throughput and never correctness. That decouples the two concerns completely: you can swap drafts, retrain them, or tune k in production without re-running an accuracy suite — which is worth a great deal operationally.

Instructor notes

Minutes: part of presenter 1's slot; instructor should insist it happens at the board. Board: The two branches, then the min + max identity. Make the room verify it sums to p. Ask the room: "Is the output the same as without speculation?" The right answer is "identically distributed, not identical — and identical at temperature 0." Expect confusion: Students assume any approximation of q degrades quality. Say: q does not appear in the answer, only in the running time.

16.4 The arithmetic: expected accepted length, and the optimal k

Model acceptance as independent with per-token probability α. The loop accepts tokens until the first rejection or until the draft is exhausted, and the target's corrected token is emitted either way — so the number of tokens emitted per verification is 1 + (accepted run length).

That is a truncated geometric series, and it is the paper's central quantity. Two sanity checks: at α → 0 it tends to 1 (speculation never helps, one token per pass), and as k → ∞ it tends to 1/(1 − α), which is the ceiling — no draft length can beat 1/(1−α) tokens per pass, so α alone caps the technique. At α = 0.8 that ceiling is 5 tokens; at α = 0.9 it is 10.

Now the cost. One cycle costs one target pass plus k draft passes, i.e. 1 + k·c target-equivalents. So, while the step remains memory-bound (§16.8 is where that fails):

Read three things off that table.

The optimum grows with α, from k = 2 to k = 7 across the rows. The reason is structural: each additional draft token costs c unconditionally and pays off only if all previous ones were accepted, with probability α^k, so speculating deeper is worth it exactly when acceptance is likely to survive the depth. A fixed k tuned for one workload is therefore wrong for another, which is what EAGLE-2's dynamic trees are for.

The curve is flat near its peak. At α = 0.8 anything from k = 3 to k = 7 is within 4% of optimal, so this parameter does not need precision — good news, since α is not known in advance and varies within a single request.

The whole technique lives or dies on α. Going from 0.6 to 0.9 nearly doubles the payoff, while choosing k well is worth a few percent. That is why EAGLE — which raises α — mattered more than any amount of tuning, and why §16.5 is about where α comes from.

Instructor notes

Minutes: 12. Protected — derive live. Board: The geometric sum, then the α = 0.8 row computed in front of them, then circle the moving optimum down the "best" column. Ask the room: "Why does the best k grow with α?" Push to: depth pays only if the whole prefix is accepted. Expect confusion: Students read the ceiling 1/(1−α) as achievable. It is a limit as k → ∞ with c = 0; at c = 0.15 the achievable peak is well below it.

16.5 Where α comes from

α is not a property of the technique or the hardware. It is a property of the draft, the target, and the text, and its variation within a single request is large.

Easy tokens are predictable by almost anything: whitespace, punctuation, closing brackets, the rest of a word already begun, boilerplate phrasing, the second half of a common idiom, and — importantly for this course — the syntax of structured output. Hard tokens are the content-bearing choices: the identifier, the number, the noun that carries the claim. So α is high in long stretches and collapses at exactly the positions that matter semantically, which is why measured acceptance is bursty and why average α understates the achievable speedup for a well-designed dynamic scheme.

Two workload observations follow, and the second is the one worth carrying to Nov 18.

Code and structured output are speculation-friendly. Both are full of syntactically forced tokens, and drafts do well on them; published acceptance rates on code tend to exceed prose.

Agent traffic should be unusually favourable, and you should measure rather than assume it. Lecture 5's ledger says an agent's prompts are 92% re-sent and its outputs are heavily templated — tool-call JSON, repeated scaffolding, the same reasoning preamble each step. All three properties are the kind that make the next token predictable. It is a reasonable expectation that agent workloads sit at the high end of α, and it is worth stating clearly that this is an expectation to test, not a published constant: the measurement is one instrumented run against your own traffic, counting accepted-run lengths, and it is the single most useful number to collect before enabling speculation in Assignment 5.

There is a subtler point about whose distribution is being predicted. The draft is trying to match the target on the target's own outputs, so a draft distilled from the target beats a same-size general-purpose model. That is why the practical recipe is a distilled or fine-tuned draft rather than "a small model you already have", and it is the seam EAGLE exploits.

Instructor notes

Minutes: part of presenter 2's slot. Board: A generated line of code with syntactically-forced tokens underlined. "α is per-token, not per-model." Ask the room: "What property of your Assignment 5 traffic would raise α?" Templated outputs, repeated scaffolding. Then: "how would you measure it?" Accepted-run-length histogram.

16.6 EAGLE: draft in feature space

EAGLE's change is to what the draft predicts, and the motivating argument is worth the reading time. Predicting the next token requires resolving all the uncertainty in one discrete choice out of 32,000. Predicting the next hidden state — the feature vector the target would have produced — is a regression problem in a continuous space, and it turns out to be substantially easier. Better still, the part that is genuinely uncertain is precisely which token gets sampled, and that can be supplied: EAGLE's draft head takes the previous feature and the token actually sampled, so it never has to guess the thing it cannot know.

The consequences are a much lighter draft — a single small head reusing the target's own hidden states rather than a separate multi-layer model — and a substantially higher α, which §16.4 showed is the lever that matters. EAGLE also drafts a tree rather than a chain, which §16.7 explains, and EAGLE-2 makes that tree context-dependent rather than fixed, spending draft budget where the model is confident.

The systems cost is real and worth pressing a presenter on: the draft is coupled to the target's internals. It consumes the target's hidden states, so it is not a black-box component you can swap; it must be trained per target model, it must run interleaved with the target rather than independently, and in a batched engine the feature plumbing has to be maintained per sequence alongside the KV cache. Compared with the original formulation — where any small model with the same tokenizer will do — EAGLE trades modularity for α. Given §16.4's table, that is usually the right trade, and it is exactly the sort of trade this course asks you to name rather than to absorb silently.

Instructor notes

Minutes: 14, presenter-led. Board (presenter): "predict token: 1-of-32,000" over "predict feature: a vector, plus you get told the sampled token". The asymmetry is the paper. Force the question: "What does feeding in the sampled token buy?" It removes the irreducible uncertainty from the regression. Instructor adds: the modularity cost. Ask: "could you use one draft for two different targets?"

16.7 Trees beat chains

A chain of k drafts commits to one continuation: if position 2 is wrong, positions 3…k are wasted regardless of how good they were. A tree hedges — propose two or three alternatives at the uncertain positions and verify all branches in the same pass, under an attention mask that lets each node attend only to its ancestors. Then accept the deepest root-to-node path that survives the acceptance rule.

Why this is nearly free is the same argument as §16.1, one level up. Verification cost is set by the weight read, which is paid once for the whole tree; adding nodes adds arithmetic, and §16.1 showed the arithmetic is idle. So at equal byte cost a tree of m nodes explores many more continuations than a chain of m.

That worked example is deliberately not a win, because the honest lesson is the one Sequoia makes formally: tree shape is an optimization problem, not a free lunch. Width helps where the model is uncertain and depth helps where it is confident, so the best tree is context-dependent — which is precisely EAGLE-2's argument — and choosing it well requires knowing the per-position confidence, which the draft can estimate. Medusa's contribution is orthogonal and worth naming: it removes the separate draft entirely, putting extra decoding heads on the target itself, which eliminates c at the cost of a lower α and a training step per target.

Instructor notes

Minutes: 8, room-led. Board: A chain of 7 nodes beside a depth-3 branching tree of 7 nodes, with the attention mask sketched (each node sees only ancestors). Ask the room: "Same number of nodes — which wins?" Then reveal that it depends on where the uncertainty is. That is the section. Expect confusion: Trees are assumed strictly better. They are strictly better per node at equal depth, and the budget is nodes, so shape matters.

16.8 When speculation becomes a net loss

Everything so far assumed the step stays memory-bound, so that extra arithmetic is free. That assumption is Lecture 2's 0.34% utilization, and it is a batch-1 statement. At batch B the step's arithmetic scales with B, and speculation multiplies it by (k+1).

The first term is unchanged by speculation — the weights are read once and each sequence's cache is read once, whatever the draft length. The second is multiplied by (k+1). So there is a batch size at which the second overtakes the first, and beyond it speculation is buying tokens by making the step slower.

State the law plainly, because it is the thing most often left out of a summary of this literature: speculative decoding converts spare arithmetic into lower latency, so it is a low-batch, latency-critical technique and it is actively harmful at the throughput-optimal operating point. An engine serving an interactive product at B = 8 should enable it; a batch-inference pipeline at B = 500 should not. And because most deployments live between those, the right implementation makes k adaptive to the current batch size — speculate deeply when the engine is quiet, not at all when it is full — which is a scheduling decision, and therefore Lecture 11's territory rather than the draft model's.

Now the inversion, which is MagicDec's contribution and the reason it is on the reading list. The memory term includes the cache, and at long context the cache dominates the weights. Redo the middle case at 16,384-token sequences, where memory permits only B = 29:

The reconciliation is clean and worth stating as a rule: what matters is not the batch size but the arithmetic intensity of the step, and long contexts push it back down by making the cache read enormous. So speculation helps whenever the step is memory-bound — at small batch for any context, and at large batch for long contexts — and stops helping when the step is compute-bound. That is one criterion instead of two, and it is Lecture 9 §9.1's three-regimes habit paying off for the last time in Part II.

Instructor notes

Minutes: 14. Protected. Board: The max() expression. Then three columns: B = 1, 164, 324, filling memory time and compute time in each. The moment compute overtakes memory is the lecture's punchline. Ask the room: "You run at B = 164 for throughput. Enable it?" Then show 1.21×. Then ask about 16K context and let the answer flip. Expect confusion: "Speculation is just free tokens." It is free arithmetic, and arithmetic stops being free at scale.

Discussion seeds

1. Identical, or identically distributed? §16.3 preserves the distribution, not the sample. Write the user-visible consequence, and say whether a product team should be told. Does your answer change at temperature 0?
2. Adaptive k. §16.8 says the right k depends on the current batch. Design the controller — what it measures, how often, and what it does when the batch is changing every iteration (Lecture 11 §11.2).
3. Measure α on your own traffic. Design the instrumentation: what you log, how you turn it into an accepted-run-length distribution, and how many tokens you need before the estimate is worth acting on (use Lecture 5 §5.3's arithmetic).
4. The modularity you gave up. EAGLE couples the draft to the target's hidden states. Price that: what does it cost when you upgrade the target model, and how would you structure a serving stack to make the cost bearable?
5. Two drafts, one target. Could an engine keep a cheap draft for high-α stretches and a better one for hard positions, switching per position? What would it need to know, and is that knowledge available before the target's pass?
6. The agent case, honestly. §16.5 argues agent traffic should have high α. Construct the experiment that would falsify it, and say what you would conclude if α turned out to be lower than for prose.

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Batch-1 decode utilization	0.34% = 1/295 — the spare FLOPs	Lecture 2 §2.13
Cost of verifying k = 4 extra tokens, B = 1	0.14 ms of arithmetic on a 4.05 ms step	§16.1
E[tokens per verification]	(1 − α^(k+1))/(1 − α), ceiling 1/(1−α)	§16.4
Speedup formula	E ÷ (1 + k·c), c ≈ 0.15 for a 1B draft	§16.4
Best speedup at α = 0.6 / 0.7 / 0.8 / 0.9	1.51× / 1.75× / 2.11× / 2.78×	§16.4
Optimal k at those α	2 / 3 / 5 / 7 — grows with α	§16.4
Acceptance identity	min(p,q) + max(0,p−q) = p	§16.3
Gain at B = 164, α = 0.8, k = 4	2.10× → 1.21× (step goes compute-bound)	§16.8
Break-even batch	B ≈ 324 — a loss above it	§16.8
Long context, B = 29 at 16K	memory-bound by 5.7× → full 2.10× restored	§16.8, MagicDec

Self-check

Exercises

1. The draft you can actually afford. You have three candidate drafts: a 1B model (c = 0.15, α = 0.75), a 350M model (c = 0.05, α = 0.62), and a Medusa-style head on the target itself (c = 0.02, α = 0.55). At batch 1, find each one's optimal k and speedup, and pick one. Solution sketch: Maximize (1 − α^(k+1))/((1 − α)(1 + kc)). 1B, α = 0.75: k = 4 gives E = (1−0.2373)/0.25 = 3.051, cost 1.60 → 1.91×; k = 3 → 1.89×; k = 5 → 1.89×. Best 1.91×. 350M, α = 0.62: k = 3 gives E = (1−0.1478)/0.38 = 2.243, cost 1.15 → 1.95×; k = 4 → 1.93×. Best 1.95×. Medusa head, α = 0.55: k = 3 gives E = (1−0.0915)/0.45 = 2.019, cost 1.06 → 1.90×; k = 2 → 1.88×. Best 1.90×. All three land within 3% of each other — the cheap-but-worse and the dear-but-better drafts nearly cancel, which is a real and underappreciated result. So choose on operational grounds, not on speedup: the Medusa head needs no separate model, no second KV cache, and no extra memory, so it wins on everything except the training step it requires per target.
2. Adaptive k against the batch. Using §16.8's step model at GQA-8 with 640-token contexts and α = 0.8, compute the best k at B = 1, 32, 128, and 256, where "best" maximizes tokens per second. Solution sketch: Memory time = (13.5 + 0.082B)/3.35 ms; compute time = (k+1)·B·13.48/494.5 ms; tokens per cycle = E(α,k)/(1 + kc) relative to baseline, but at large B the binding term is compute. B = 1: memory 4.05 ms dominates for all k ≤ 8, so take the unconstrained optimum k = 5 → 2.11×. B = 32: memory 4.81 ms; compute at k = 5 is 6 × 32 × 0.02726 = 5.23 ms — just compute-bound; k = 4 gives 4.36 ms, still memory-bound, E = 3.36, net 2.10×. Take k = 4. B = 128: memory 7.16 ms; compute at k = 2 is 3 × 128 × 0.02726 = 10.5 ms, already compute-bound. Even k = 1: 6.98 ms, memory-bound, E = 1.8, cost 1.15 → 1.57×. Take k = 1. B = 256: memory 9.30 ms; k = 1 compute 2 × 256 × 0.02726 = 13.96 ms → compute-bound, gain = 1.8 × 9.30/13.96 / 1.15 = 1.04× — barely worth it. Take k = 1 or disable. The controller is monotone and steep: k falls from 5 to 1 between B = 1 and B = 128, which is why a fixed k is wrong for any engine whose batch varies.
3. Sizing the α measurement. You want α to ±0.02 with 95% confidence before choosing k. Treating each drafted token as a Bernoulli trial, compute the number of drafted tokens required, and then the number of requests at 128 output tokens each and k = 4. Solution sketch: Half-width 1.96√(α(1−α)/n) ≤ 0.02 at α ≈ 0.8 → √(0.16/n) ≤ 0.0102 → n ≥ 0.16/1.041e-4 = 1,537 drafted tokens. Each verification drafts k = 4 tokens and emits E = 3.36, so a 128-token response involves 128/3.36 = 38.1 verifications = 152 drafted tokens. So ≈10 requests suffice for ±0.02 — the estimate is cheap, which is the point. But note the independence assumption is wrong: acceptance is bursty and autocorrelated within a response (§16.5), so the effective sample size is smaller than the token count. Inflate by a factor of 3–5 for safety and measure 30–50 requests, which is still trivial. There is no excuse for assuming α rather than measuring it, which is the exercise's real content.
4. When does the draft's cache matter? The draft keeps its own KV cache. For a 1B draft at proportional architecture (roughly 1/6.74 of the target's per-token KV), compute the extra memory at B = 164 with 640-token contexts under GQA-8, and the concurrency it costs. Solution sketch: Target per-token KV 128 KiB; draft ≈ 128/6.74 = 19 KiB/token. At 640 tokens that is 0.0122 GB per sequence against the target's 0.082 GB — a 14.9% increase in per-sequence cache. Concurrency at the 62.5 GB budget falls from 762 to 62.5/(0.082+0.0122) = 663, i.e. 13% fewer sequences. At the SLO-capped B = 164 the memory is not binding, so the cost is invisible there — but the step bytes rise from 26.9 to 28.9 GB, adding 0.6 ms (7.5%) to the memory term, which slightly worsens §16.8's break-even. So the draft's cache is a second-order cost that matters only when memory binds — and it is one more argument for Medusa-style heads, which have no separate cache at all.
5. Compose with the rest of Part II. Apply Lecture 14 §14.7's 25.6× cache reduction (20 KiB/token) and Lecture 15's 4-bit weights (3.47 GB) to §16.8's break-even calculation, at α = 0.8, k = 4, 4,096-token contexts. Find the new break-even B and explain the direction of the change. Solution sketch: Per-sequence cache = 4,096 × 20 KiB = 0.082 GB. Memory time = (3.47 + 0.082B)/3.35 ms. Compute is unchanged at 5·B·13.48/494.5 = 0.1363B ms (dequantization aside). Break-even when 3.36 × (3.47 + 0.082B)/3.35 = 0.1363B → 1.0031(3.47 + 0.082B) = 0.1363B → 3.481 + 0.0822B = 0.1363B → B = 64. The break-even collapsed from 324 to 64, and the reason is instructive: every successful byte optimization in Part II shrinks the memory term, which is the very term that was giving speculation its free arithmetic. The techniques are in tension — quantization and cache reduction make the step more compute-bound, which is exactly the regime where speculation stops working. That is the sharpest example in the course of two wins that do not compose, and it is why an engine must decide these jointly rather than enabling every flag.

Reading guide

Required — Speculative decoding. Short, and the rare paper where the appendix is the point. Read §2 and the proof until you can reproduce §16.3's two-branch argument on a whiteboard; if you can state why q does not appear in the result, you have the paper. Then their analysis of expected accepted length — check it against §16.4's geometric sum — and their walltime model, noting which of its assumptions is the one §16.8 breaks. Skip the machine-translation experiments. Hold this question: the guarantee is distributional. Write down precisely what a user could and could not detect, and decide whether "exact" is the right word to use with a product team.

Required — EAGLE. Read the motivating analysis first and slowly — the claim that feature-space prediction is easier than token-space prediction, and that the irreducible uncertainty is the sampled token, which you can simply provide. That argument is the whole paper and everything else is its consequence. Then the draft head and the tree. Read the acceptance-rate tables as the result, and cross-check against §16.4: given their α, what speedup does the formula predict, and does it match what they report? A gap tells you something about c that the prose may not. Hold this question: the draft consumes the target's hidden states — enumerate what that couples, and what it would cost your serving stack when the target model is upgraded.

Optional — Medusa. No separate draft: extra heads on the target, verified under a tree mask. Read the tree-attention construction, which is §16.7's mechanism in its clearest form. Question: it eliminates c and lowers α — using §16.4's formula and Exercise 1, does it win?

Optional — Sequoia. Tree construction as an optimization problem, with hardware-awareness. The best optional paper if §16.7's inconclusive worked example annoyed you. Question: their optimum depends on hardware — which quantity from §16.8 enters, and does their answer change at B = 164?

Optional — EAGLE-2. Context-dependent dynamic trees. Question: §16.5 says α varies within a single response — how much of EAGLE-2's gain is just exploiting that variance, and could a simpler rule capture most of it?

Optional — MagicDec. The most important optional reading today, because it contradicts the folk summary of the technique. Read the analysis of where the bottleneck sits at long context and check it against §16.8's two worked cases. Question: they argue speculation helps at large batch for long sequences; restate their condition in terms of arithmetic intensity, and say what it becomes after Lecture 14's 25.6× cache reduction (Exercise 5).

Looking ahead

Wednesday (agent serving systems I, Nov 18) ends the run of single-request optimizations and changes the unit of work. Every lecture from Lecture 10 to today has made one request cheaper; Parrot's observation is that the thing your users actually run is a program that issues hundreds of dependent requests, and that the serving system is told none of its structure — so it cannot see that step 7 depends on step 6, that twenty branches share a prompt, or that only the last output is latency-critical. Autellix and Teola follow in the second half of that meeting, and Nov 23 takes the stalls, the session state, and the multi-agent case.

Speculation is the right bridge into that block for two reasons worth holding. First, §16.5's expectation that agent traffic is unusually speculation-friendly — templated outputs, repeated scaffolding — is a hypothesis that the agent-serving lectures give you the vocabulary to test properly. Second, §16.8's law is about whose spare capacity you are spending, and an agent workload spends its life waiting on tool calls (Nov 23 prices the stall at half the wall-clock), which means the engine is often idle in precisely the way speculation exploits.

Assignment 5 goes out today and is due Dec 2. If you enable speculation in A5, §16.8 is the section to quote: report the batch size you measured at, because a speedup without it is uninterpretable.

One thing to carry out of the room. Speculation is the only optimization in Part II that is exactly output-preserving and attacks the floor rather than the bytes — and its reward is bounded by how idle your machine is. Every other lecture made the machine busier. The last two lectures of Part II therefore work against each other, and noticing that before you turn on both flags is the difference between a 2.1× and a 1.04×.