Part I ended with a bill and Part II starts with the machine that pays it. Lecture 2 left one number on the board: an H100's decode arithmetic wants a batch of 295 and its KV cache permits 29. That is the gap the next two months attack, and today is the first attack — not on the arithmetic, which is fixed, but on the allocator, which turns out to be throwing away most of the 29 before any scheduler gets a chance to be clever. The claim is deliberately unglamorous: the largest single win in LLM serving came from noticing that the KV cache had been implemented as one contiguous array per request, and that this is the same mistake operating systems stopped making in 1962. By the end you should be able to compute how much memory a reserve-to-maximum allocator wastes on a given workload, explain what a block table costs and what it buys, derive the swap-versus-recompute threshold for a preempted sequence, and say which of the semester's remaining problems paging solves, which it merely makes solvable, and which it does not touch at all.

Where this sits

Part I priced the workload from the outside: Lecture 3 counted tokens, Lecture 4 wrote the loop, Lecture 5 built the instrument, and the bridge out of Part I stated the four properties of an agent request stream — 92% re-sent prompt tokens, prefill-dominated compute, strong dependence between requests, and sessions that stall for seconds holding gigabytes. Lecture 2's serving half then priced the machine that receives that stream and found it structurally underfed.

Today is the floor of Part II: what a serving engine actually is, and the one design decision inside it that determines whether everything above it has room to work. The order matters. Paged allocation comes first because iteration-level scheduling (Oct 7) needs to add and remove sequences from a running batch without repacking memory; because prefix sharing (Nov 9) needs KV blocks that can be referenced rather than owned; and because disaggregation (Oct 14) needs a cache that can be described by a table small enough to send over a network. Paging is not the most interesting idea in Part II. It is the one the interesting ideas are standing on.

Instructor notes — Timing plan

75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221). Instructor-led, one paper. This is the opening of Part II, so five of the seventy-five minutes are spent on where the semester is going.

Time	Segment	Notes
0–6	Part II opens: 29 against 295 (§10.1)	Re-board Lecture 2's collision, then the five jobs of an engine. Do not start the paper yet.
6–16	§10.2 The naive allocator	Protected. Derive 14% utilization live. The room must feel that 86% of the cache is empty.
16–36	§10.3 PagedAttention (paper deep-dive)	Protected — never cut. Block table on the board; then internal fragmentation at 1.3%, and the kernel's new job.
36–46	§10.4 What paging buys next: sharing	Copy-on-write, the 5.7× fan-out, and one sentence pointing at Nov 9.
46–58	§10.5 The engine loop, admission, preemption	The 67 ms swap versus 129 ms recompute derivation is ours — do it live.
58–68	§10.6 Sizing and the SLO ledger	TTFT decomposition; then goodput, and why a mean is a lie.
68–75	§10.7 The map of Part II + close	Point at the eight dates. End on "the allocator was the easy 7×."

Reading-only, not scheduled: §10.7's table in full, and the block-table memory overhead arithmetic in §10.3 (state the 2 KB result, assign the derivation).

If running long: compress §10.6 to the TTFT decomposition and drop the goodput table; cut §10.4 to the copy-on-write picture and the 5.7×. Never cut §10.2 or §10.3 — the waste measurement and the block table are the lecture.

Learning objectives

By the end of this class you should be able to:

1. List the five jobs a serving engine performs on every request, and name the resource each one contends for.
2. Compute the memory a reserve-to-maximum allocator wastes for a given distribution of prompt and output lengths, and convert that waste into lost concurrency and lost throughput.
3. Distinguish internal from external fragmentation in a KV cache, and say which one paging eliminates and which one it merely bounds.
4. Draw the block table for a batch of sequences, compute its memory overhead, and state what the attention kernel must now do that it did not have to do before.
5. Explain copy-on-write sharing for parallel sampling, and compute the sharing factor for a fan-out of n completions off a common prompt.
6. Derive the swap-versus-recompute threshold for a preempted sequence from its KV bytes, a link bandwidth, and its recompute cost — and say why that threshold is a property of the model and the machine rather than of the request.
7. Decompose a TTFT budget into queueing, prefill, and scheduling delay, and explain why goodput and throughput can move in opposite directions.
8. Place each remaining Part II topic against the specific quantity in today's arithmetic that it attacks.

10.1 What a serving engine is asked to do

A serving engine is the thing between an HTTP request and 201 forward passes. It has five jobs, and each one contends for a different resource, which is why they are usually five different pieces of code that fight with each other.

Admit. Decide whether to accept a request now, queue it, or reject it. The resource is memory, not compute: admitting a request means promising it KV space for as long as it lives, and the engine cannot know how long that is because it does not know how many tokens the model will emit. Admission in an LLM engine is therefore a bet on an unknown length, which is the root of most of its scheduling pathology.

Allocate. Find KV space for the sequence's prefill and for its growth. This is today's subject.

Batch. Choose which sequences run in the next forward pass. Lecture 2 §2.14 says the answer should be "as many as possible"; §2.15 says memory will not allow it; Oct 7 says the batch should be re-chosen every iteration rather than every request.

Run. Launch the kernels — prefill for arrivals, decode for everyone else, and, if the engine is willing, both in the same step. §2.16's 6.7× interference spike lives here, and Oct 5 is about the kernels themselves.

Stream. Return tokens as they are produced, which means the engine's per-step latency is directly a user-visible property and not an internal detail.

Two of those five are about memory and only one is about arithmetic. That ratio is the whole character of LLM serving, and it is why an engine's most consequential internal component is its allocator.

Instructor notes

Minutes: 6, and the first two are Part II's framing rather than today's. Board: "29 against 295" from Lecture 2, then the five verbs in a column with the contended resource beside each. Circle memory twice. Ask the room: "Which of the five can the engine get wrong in a way it cannot recover from?" Admit — because the promise has already been made. This lands §10.5 forty minutes early. Expect confusion: Students expect a serving system to be a scheduling problem. Say: "It is a memory-management problem wearing a scheduler's clothes."

10.2 The naive allocator, and where 86% of your cache goes

Here is how a KV cache gets implemented the first time, by everyone, including every framework that shipped before 2023. A sequence's keys and values are a tensor. Its shape must be known when it is allocated. The sequence will grow to an unknown length, and moving a multi-gigabyte tensor mid-flight is unthinkable, so you allocate for the maximum length the request is permitted — S_max — and let the sequence grow into it.

This is correct, simple, contiguous, and kernel-friendly. It is also catastrophic, and the reason is one line of arithmetic: the allocation is sized by the limit and the occupancy is set by reality.

Sit with the shape of that result, because it is better than it looks. Lecture 2 presented 29 as a hardware fact and the gap to 295 as a wall. It is not a wall; it is two walls, one of which is made of software. Packing the cache takes the workload above from 29 to 200 concurrent sequences, which is 68% of the way to the ridge point — and it costs nothing but a data structure.

Name the two wastes precisely, because paging treats them differently and the distinction is on the self-check.

Internal fragmentation is space reserved inside a sequence's own allocation that the sequence has not filled: the 3,456 unused tokens of a 4,096-token slot occupied by a 640-token sequence. It is 84% of the budget above, it is proportional to S_max − S_actual, and it is what makes long context ruinous — raise S_max to 32,768 to advertise long context and the same chat workload's utilization falls to 1.9%, because the slot grew 8× and the occupancy did not.

External fragmentation is space between allocations, too small for the next request. With uniform 2.147 GB slots there is none, which is the one virtue of reserving to maximum; permit per-request S_max values and it reappears immediately, as a 1.2 GB hole that no 2.147 GB slot fits.

There is a third waste that no allocator can remove, and it is worth naming so that today's win is not oversold. A sequence that will emit 128 tokens has 127 of them un-emitted at the start; the space for them must be available even if it is not yet occupied. Paging makes that space available to whoever needs it right now instead of dedicating it, which is exactly the trade a demand-paged OS makes, along with the same risk: you can hand out promises you cannot keep. §10.5 is what happens when you do.

Instructor notes

Minutes: 10. Protected. Board: Draw one 4,096-token slot as a long rectangle, shade the first 640. Then draw 29 of them stacked. Then write "9.5 of 62.5". Only then say 15%. Ask the room: "What is the utilization if we raise S_max to 32K to advertise long context?" Let them compute 1.9%. This is the moment the paper's motivation becomes obvious rather than asserted. Expect confusion: "Just set S_max per request." Good instinct, wrong outcome — you have traded internal fragmentation for external, and you still cannot grow a sequence past its own guess. Say it in one sentence and move on; it is Exercise 2. Common wrong answer: "Allocate lazily and realloc when it grows." Copying 2 GB mid-decode, per sequence, while 28 others wait on the step. Price it out loud.

10.3 PagedAttention: the KV cache as virtual memory

The fix is the oldest idea in operating systems, and the paper says so in its title. Stop requiring a sequence's KV to be contiguous. Cut the cache into fixed-size blocks of b_blk tokens, keep a pool of free blocks, and give each sequence a block table — an array mapping its logical block index to whichever physical block happens to hold it. A sequence grows by acquiring one more block when its last one fills. It never reserves for a length it has not reached, and it never needs its blocks to be neighbours.

That is the whole mechanism. Everything interesting follows from it, in three steps.

Step one: internal fragmentation collapses from S_max − S_actual to less than one block. A sequence wastes only the unfilled tail of its final block, at most b_blk − 1 tokens and on average about b_blk/2, regardless of S_max. S_max stops being a memory parameter and becomes what it should always have been: a policy limit.

Step two: the block size is a real tradeoff, and it is the one knob here worth tuning. Small blocks waste less tail but lengthen the block table, cost more per-step gather bookkeeping, and give the attention kernel shorter contiguous runs to work with. Large blocks are kernel-friendly and waste more. Sixteen is a default, not a law:

b_blk	Mean tail waste (MHA)	Waste at B = 199	Table entries at 4K	Kernel's contiguous run
1	0	0%	4,096	1 token — pathological
16	8 tokens = 4.0 MB	1.3%	256	16 tokens
64	32 tokens = 16.0 MB	4.8%	64	64 tokens
256	128 tokens = 64.0 MB	17.0%	16	256 tokens

Read the last two columns together: block size trades memory against kernel efficiency, and the right answer depends on the sequence lengths and on how good your gather kernel is. It also interacts with sharing — §10.4's copy-on-write duplicates a whole block on a single divergent token, so large blocks make forks expensive. This is why FlashInfer exists and why Oct 5 follows this lecture rather than preceding it.

Step three: the cost did not vanish, it moved into the kernel. A contiguous cache lets attention read S tokens of keys as one strided load. A paged cache makes the kernel take a block table and gather. The arithmetic is unchanged — the FLOPs and the bytes are identical, so §2.15's 1 FLOP/byte for MHA attention still holds — but the access pattern is now indirect, and a naive gather can lose enough bandwidth to eat the win. PagedAttention is therefore two things wearing one name: an allocator and a fused attention kernel that reads through a block table at close to contiguous speed. The reason this idea arrived in 2023 rather than 2019 is not that nobody thought of paging; it is that the kernel had to be written.

Instructor notes

Minutes: 20. Never cut. This is the paper. Board: Two sequences, logical blocks left, one shared physical pool right, arrows crossing. Then grow sequence A by one token and let the room tell you what happens (new block, one table entry). Then the 15.2% → 98.7% pair, then 29 → 199. Ask the room: "What does the attention kernel now have to do?" Push until someone says gather. Then: "So where did the cost go?" This is the sharpest question of the lecture. Expect confusion: Students think paging reduces KV bytes. It does not — 512 KiB per token is unchanged. It reduces reserved-and-unused bytes. Say it twice; it is self-check 3. Common wrong answer: "Block size 1 is optimal, waste is zero." Let them see the table's last column and the 4,096-entry table.

10.4 What paging buys after capacity: blocks can be shared

Capacity was the motivation; sharing is the part that reshapes the rest of the semester. Once a sequence's KV is a list of references to blocks rather than an owned array, two sequences can name the same block. Reference-count it, and copy only on divergence, and you have copy-on-write for KV.

The immediate application is one request that wants n completions from one prompt — parallel sampling, best-of-n, beam search, or an agent fanning a subtask across n attempts, which is Lecture 5's k ≈ 6 crossover arriving as a memory question. The prompt's blocks are identical for all n branches by construction, so allocate them once.

Only prefill is shared here, and only the exactly-identical prefix, and only within one request. Two observations follow, and they are the two halves of the rest of Part II.

First, the sharing is exact, not approximate. Blocks are shared because the tokens are identical, so the K and V values are identical — the same theorem the Nov 9 lecture proves properly. No output changes. This keeps paging in the family of optimizations that need no accuracy caveat, which is why it is on by default in every engine.

Second, nothing about the mechanism cares that the branches came from one request. If two different requests from two different users begin with the same 2,000-token system prompt, their first 125 blocks are identical and could be shared just as soundly. vLLM's own §4.3 stops at intra-request sharing; noticing that the block pool is already a content-indexable cache, and building the index and the eviction policy to exploit it, is exactly what Nov 9 does — and a final project that designs one has a ready-made baseline. Today's takeaway is narrower and load-bearing: the data structure that made the cache fit is the same data structure that makes the cache shareable. That is not a coincidence, it is the usual reward for using the right abstraction.

Instructor notes

Minutes: 10. Board: One prompt's blocks, eight branch arrows pointing at them, refcount 8. Then one branch writes a token: new block, refcount 7 + 1. Then 17.9 GB against 3.2 GB. Ask the room: "Does anything in this mechanism know the branches came from the same request?" No. Let the silence do the work, then name Nov 9. Expect confusion: Beam search gets treated as the motivating case. It is the historical one; the live one is agent fan-out and shared system prompts.

10.5 The engine loop, admission, and the two ways to break a promise

Assemble the pieces into the loop an engine actually runs. Every iteration: pick the set of sequences to run, allocate any blocks they need to grow, run one forward pass, append the emitted tokens, retire anyone who finished, and admit new arrivals into the freed space.

Two things about that loop are worth stating now, because they are what Oct 7 builds on.

The batch is re-chosen every iteration, not every request. A finished sequence leaves at the end of the step that finished it, and its blocks are available to the next arrival immediately, rather than when the slowest member of some fixed batch completes. Paging is what makes this cheap: adding and removing a sequence is editing a table, not repacking a tensor. Request-level batching wastes the tail of every batch on whichever sequence is longest; iteration-level batching does not. That is Orca's contribution and it is the Oct 7 lecture — today only note that it requires today's allocator to be practical.

Admission is a promise the engine can be forced to break. The engine admits a sequence knowing its prompt length and not its output length. Blocks are handed out on demand, so a card can be fully admitted and still run out when the resident sequences all decide to keep going. Something has to give, and the engine has exactly three moves: refuse new arrivals (safe, hurts TTFT and throughput), preempt a resident sequence, or crash. Real engines do the first two and reserve a watermark of free blocks so that the second is rare.

Preemption is the interesting one, because it has two implementations and choosing between them is an arithmetic problem you can do from Lecture 2's numbers alone. Evict a sequence's KV and you must either swap it to host memory and bring it back, or discard it and recompute the prefill when the sequence is rescheduled.

Three things in that box deserve to be said out loud. First, the threshold is a property of the model and the machine, not of the request: every 4K MHA sequence on this card has the same 67 ms against 129 ms, so the policy can be decided once, offline. Second, GQA changes the answer, not just the magnitude — shrinking the cache makes swapping cheap while leaving recompute alone, so an architectural decision taken at pretraining time silently rewrites the engine's preemption policy. Third, both numbers are stalls: 67 ms is nearly three decode steps at the 22.6 ms ambient TPOT, so preemption is visible to the user whichever branch you take, and the right engineering goal is to preempt rarely rather than to preempt cheaply.

This exact calculation returns twice more, which is a good sign the framing is right. On Nov 9 it becomes fetch-versus-recompute against a remote KV store, and on Nov 23 it becomes keep-versus-swap-versus-discard for a session stalled on a tool call — where the same 2.147 GB and the same 67 ms reappear, because a 16,384-token GQA-8 session holds exactly the bytes a 4,096-token MHA sequence does.

Instructor notes

Minutes: 12. Board: The loop as six verbs in a cycle. Then the watermark idea. Then derive 67 against 129 live — do not show the result first. Ask the room: "The engine admitted 199 sequences and they all keep generating. What breaks?" Blocks run out. Then: "Which of your three moves is cheapest for the user?" Refusing new arrivals — nobody resident is hurt. That is why watermarks exist. Expect confusion: Preemption is assumed to lose the sequence's progress. It loses its cache, never its tokens — the tokens are the state, and Lecture 4 §4.5 said so. If short on time: State the 67/129 pair and assign the break-even bandwidth as Exercise 4.

10.6 Sizing, SLOs, and where the time actually goes

Paging changed capacity. Turn capacity into a service level, because that is what an operator is actually asked for, and because Assignment 4 will ask you for exactly this ledger.

TTFT decomposes into three terms, and only one of them is the model. Queueing — how long the request waits for blocks and a slot, which is the admission policy's output and, on a loaded engine, usually the largest term. Prefill — 129 ms for 4,096 tokens at 50% of peak, the only term Lecture 2 priced. Scheduling delay — how long until the next iteration boundary, at most one step, so ≤22.6 ms here. An engine that reports "129 ms TTFT" is reporting the term it controls least.

TPOT is a property of the batch, not of the request, which is the single most counter-intuitive thing about serving. Lecture 2 §2.15: a step moves 13.5 GB of weights plus B × KV-per-sequence, so your latency depends on how many strangers are generating alongside you. At B = 199 with 640-token sequences that is 13.5 + 199 × 0.328 = 78.8 GB → 23.5 ms. Your neighbours set your cadence.

Which is why goodput is the metric and throughput is not. Goodput counts only requests that met their SLO. Give the middle column a TPOT SLO of 20 ms and its 8,470 tok/s becomes zero goodput — every request violates — while the naive column, at a third of the throughput, satisfies all of them. Neither configuration is wrong; the question "what is this engine's throughput?" is wrong, because throughput without a latency constraint is a number you can always increase by making the service worse. The correct operating point is the largest B whose TPOT clears the SLO, which for a 20 ms target and 0.328 GB sequences is (20 ms × 3,350 GB/s − 13.5 GB) ÷ 0.328 GB = 164 — so the memory ceiling of 199 is not even the binding constraint here. After paging, the SLO usually binds before the memory does. That is the clearest sign the allocator problem is genuinely solved, and it is why the next eight weeks are about scheduling, kernels, and bytes rather than about allocation.

Instructor notes

Minutes: 10. Board: TTFT as three stacked bars with queueing biggest. Then the three-column table, then strike out the middle column's throughput and write "goodput 0". Ask the room: "Which column would you deploy?" Force them to ask for the SLO first. If nobody does, that is the lesson. Expect confusion: "Paging made latency worse, so it is a wash." No — it moved the operating point along a frontier you could not previously reach. At B = 164 you get both.

10.7 What paging did not fix: the map of Part II

Today removed one term from one equation. Put the rest of the semester against what remains, because every date below is a specific quantity in today's arithmetic that is still wrong.

Still broken after today	The quantity	Where it goes
The gather kernel paging created, and attention's 1 FLOP/byte	achieved bandwidth on an indirect access	GPU programming Sep 30, kernels Oct 5
The batch is re-chosen per request, wasting every batch's tail	effective B over time	Batching and scheduling I, Oct 7
One 129 ms prefill wrecks 199 decodes' cadence	§2.16's 6.7× TPOT spike	Disaggregation, Oct 14
A cache hit on the wrong replica is a cache miss	which card a request lands on	Routing and load balancing, Oct 19
512 KiB per token, paid in full for every position	KV bytes per token	KV-cache optimization, Oct 28
Identical prefixes across requests are prefilled twice	the 92% re-send of Lecture 5	Prefix cache, Nov 9
13.5 GB of weights re-read every step, b = 2	bytes per weight and per cached value	Pruning and quantization, Nov 11
One forward pass yields one token	tokens per pass over the weights	Speculative decoding, Nov 18
The engine sees finished strings, not the agent's structure	the dependence between requests	Agent serving, Nov 18 and Nov 23

Two honest closing observations. Paging was, in a real sense, the easy 6.9× — a known solution to a known problem, findable by anyone who had taken an operating systems course and looked at where the memory went. It is also the last time in this course that a single idea will buy that much, which is why the remaining lectures are about compounding smaller factors and about the tradeoffs between them. And the reason it was available to be found is worth internalizing: nobody had measured the utilization. The paper's contribution was a measurement first and a mechanism second — which is the same claim Lecture 5 made about your own agent, one layer up.

Discussion seeds

1. The knob you were given. vLLM exposes block_size. Using §10.3's table and your Assignment 2 agent's actual sequence-length distribution, what would you set it to, and what measurement would change your mind?
2. Admission without lengths. Suppose you had a perfect predictor of output length. Which of §10.5's three moves would you stop needing, and what new failure would a wrong prediction cause?
3. Where the analogy breaks. LLM paging has no TLB and no demand paging. Which of those two absences costs more, and what would a KV "TLB" even be?
4. Utilization as the claim. The paper leads with memory utilization rather than throughput. Construct a workload on which utilization goes from 15% to 99% and throughput barely moves. What does your construction tell you about when to deploy this?
5. The fan-out you will actually run. Lecture 5 put the single-context-versus-fan-out crossover at k ≈ 6 on token cost alone. Redo it with §10.4's sharing factor in hand — does cheap fan-out move the crossover, and should it change how you write Assignment 3?

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Naive-allocator utilization, 640-token workload at S_max = 4K	15.2% (9.5 of 62.5 GB)	§10.2
Same, at S_max = 32K	1.9%	§10.2
Concurrency: naive → paged	29 → 199 (6.9×)	§10.2, §10.3
Internal fragmentation, b_blk = 16	≤15 tokens, mean 8 → 1.3% of budget	§10.3
Block-table overhead, 4K sequence	256 entries ≈ 2 KB, per 2.1 GB of KV	§10.3
Fan-out sharing, n = 8 off a 4K prompt	5.7× (34,816 → 6,144 tokens)	§10.4
Fan-out, n = 16 off 8K	73.0 GB → 8.6 GB (8.5×); unshared does not fit	§10.4
Swap round trip, 4K MHA at 64 GB/s	67 ms (2.147 GB each way)	§10.5
Recompute, 4K prefill at 50% peak	129 ms (55.2 + 8.8 TFLOP)	§10.5, Lecture 2 §2.16
Swap/recompute break-even bandwidth	33.3 GB/s MHA, 8.3 GB/s GQA-8	§10.5
TPOT at B = 199, 640-token sequences	23.5 ms (78.8 GB/step)	§10.6
Largest B meeting a 20 ms TPOT SLO	164 — below the memory ceiling of 199	§10.6

Self-check

Exercises

1. The long-context brochure. Your service advertises a 128K context window, so S_max = 131,072. The real workload is unchanged: 512-token prompts, 128-token outputs. Under a naive allocator, compute the slot size, the concurrency, the utilization, and the paged concurrency for comparison. Then say what the marketing claim actually cost. Solution sketch: Slot = 131,072 × 512 KiB = 68.7 GB, which exceeds the 62.5 GB budget, so the card serves zero requests — the feature is unimplementable under this allocator at MHA. Relax to GQA-8: slot = 17.2 GB, concurrency 3, occupancy 640 × 128 KiB = 0.082 GB, so utilization = 3 × 0.082 ÷ 62.5 = 0.39%. Paged with GQA-8: 62.5 ÷ (0.082 + 0.002) ≈ 744 sequences. The brochure claim cost a factor of 248× in concurrency, and it cost it for every request, including the 99% that never exceeded 1,000 tokens. Under paging the claim is nearly free, because S_max stops being a memory parameter — which is why long-context serving and paged allocation arrived together.
2. Per-request S_max, and the fragmentation you invited. Instead of paging, you let each request declare its own S_max. Requests arrive with S_max ∈ {512, 4,096, 32,768} in proportions 70/25/5. Compute the mean slot size, the mean concurrency, and then describe the failure mode that paging does not have. Solution sketch: Slots are 0.268, 2.147, and 17.2 GB; mean = 0.7·0.268 + 0.25·2.147 + 0.05·17.2 = 1.586 GB, so mean concurrency ≈ 62.5 ÷ 1.586 = 39 — better than 29, and still far under 199. The new failure is external fragmentation: retire a 512-token sequence and you have a 0.268 GB hole that no 2.147 GB request can use, so the card can hold 20 GB free and still refuse a medium request. Worse, the sequence still cannot grow past its declared S_max, so you have added an admission-time guess that is wrong in both directions — truncate the ones that under-guessed, waste memory on the ones that over-guessed. Variable-size allocation converts a quantity problem into a placement problem; fixed-size blocks are how every allocator since 1962 has refused that trade.
3. Pick a block size properly. Your measured sequence-length distribution is lognormal with median 800 tokens and 5% of sequences over 20,000. For b_blk ∈ {8, 16, 64, 256}, compute the mean tail waste per sequence (MHA), the total waste at B = 199, and the block-table entries for a median and a p95 sequence. Then state the two measurements you would need before choosing. Solution sketch: Mean tail ≈ b_blk/2 tokens → 2, 8, 32, 128 tokens → 1.0, 4.0, 16.0, 64.0 MB per sequence; at B = 199, 0.20, 0.80, 3.18, 12.7 GB, i.e. 0.3%, 1.3%, 5.1%, 20.4% of the 62.5 GB budget. Table entries at 800 tokens: 100, 50, 13, 4; at 20,000: 2,500, 1,250, 313, 79. So on memory alone, 8 or 16. The two measurements that decide it: (a) achieved attention bandwidth as a function of contiguous run length on your kernel — if 16-token runs cost 20% bandwidth against 64-token runs, the 3.8% extra memory buys it back and then some; (b) your fork/branch rate, since copy-on-write duplicates a whole block on one divergent token, so b_blk = 256 makes a fan-out of 8 copy 8 × 64 MB = 512 MB it did not need to. The memory column alone picks the wrong answer, which is the point of the exercise.
4. The break-even you will actually hit. Derive the swap-versus-recompute break-even bandwidth as a formula in S, n_kv, and the achieved prefill rate. Evaluate for a 16K GQA-8 sequence, and then find the sequence length at which swapping stops winning on a 32 GB/s link under MHA. Solution sketch: Swap = 2·(2·L·n_kv·d_head·b·S) ÷ W; recompute = (2N·S + 4·L·n_heads·d_head·S²) ÷ F. Break-even W* = 2·KV·F ÷ (2N·S + 4·L·n_h·d_h·S²). Note the numerator is linear in S and the denominator is quadratic, so W* falls as S grows — recompute gets relatively worse with length, because attention is quadratic while the cache is linear. For 16K GQA-8: KV = 2.147 GB, swap = 67 ms; recompute = 220.9 + 140.7 = 361.6 TFLOP ÷ 494.5 = 731 ms; swap wins 10.9×, break-even W* = 5.9 GB/s. For MHA at 32 GB/s, swap = 2·S·512 KiB ÷ 32e9 and recompute = (13.48S + 0.524S²·1e-3) GFLOP ÷ 494.5e12; setting them equal gives S ≈ 2,600 tokens — below that, the link is too slow and you should recompute; above it, swap. So a single global preemption policy is wrong for a mixed-length workload, and the crossover is computable per sequence from its length alone.
5. What paging did and did not buy, on the agent ledger. Take Lecture 5's 20-step agent: 97,000 submitted prompt tokens, 7,700 distinct, ~80 output tokens per step. Under GQA-8, compute the peak KV a single such session holds, how many sessions a card holds, the prefill FLOPs the session costs, and then state which of those three numbers paging improved. Solution sketch: The session's context grows to 7,700 tokens, so peak KV = 7,700 × 128 KiB = 0.985 GB; the card holds 62.5 ÷ 0.985 ≈ 63 such sessions concurrently (and 29 of them if you had reserved to a 16K S_max, so paging is worth 2.2× here, less than the 6.9× of §10.3 because agent contexts genuinely are long). Prefill: 97,000 × (13.48 + 8.59·(S/16,384)) — using Lecture 2's flat weight term alone, 97,000 × 13.48 GFLOP = 1.31 PFLOP ÷ 494.5 TFLOP/s = 2.64 s of exclusive H100 time per task. Paging improved the second number and did nothing to the third — and the third is 92% waste. That is the whole argument for Nov 9: the allocator made the cache fit, and the cache is still being recomputed from scratch every step.

Reading guide

Required — vLLM / PagedAttention. Read §3 first and read it as a measurement paper: the memory-waste breakdown is the contribution, and everything after it is engineering in service of a number somebody bothered to collect. Then §4, until you can draw the block table for two sequences sharing a prompt and say exactly what happens on a fork — that picture is the one you will reuse on Oct 7, Oct 28, and Nov 9. §4.3 on sharing is short and is the seed of the Nov 9 lecture; do not skim it. Read §4.4 on preemption against §10.5's arithmetic and check whether their all-or-nothing eviction policy matches the break-even you computed. Skim §5's implementation and §6's evaluation, and be disciplined about the headline multipliers: they are measurements on specific workloads with specific length distributions, and §10.6's table shows how much a throughput ratio moves when the latency constraint changes. Hold this question: the paper's claim is a utilization figure, but the reason anyone cares is throughput — construct the workload on which utilization improves enormously and throughput does not, and you will understand exactly what this technique is for.

Optional — FlashInfer. The sequel to today's third step: once the cache is paged, attention is a block-sparse gather, and someone has to make that fast for every combination of block size, head layout, and quantization. Read the format abstraction and the JIT/scheduler sections; skip the microbenchmark tables. Question: which of §10.3's tradeoffs does a sufficiently good kernel make irrelevant, and which one survives any kernel?

Optional — vLLM docs. Read for Assignment 4 rather than for the lecture. Find gpu_memory_utilization, max_num_seqs, max_num_batched_tokens, and block_size, and write one sentence for each mapping it to a quantity in these notes — respectively the 62.5 GB budget, the B of §10.6, the chunked-prefill limit that Oct 14 will explain, and §10.3's table. If you can do that, you can configure an engine deliberately instead of by superstition, which is most of what Assignment 4 is graded on.

Looking ahead

Wednesday (GPU programming, Sep 30) goes down a layer to the machine all of this runs on — the memory hierarchy, occupancy, and why §10.3's gather is a hardware question — and Oct 5 makes it a kernel, with the roofline paper as required reading and FlashAttention as the worked example of turning §2.3's byte ratio into a 10× that paging cannot touch. Then Oct 7 takes today's loop and re-chooses the batch every iteration, which today's allocator is what made possible.

Assignment 2 (design an agent) is due Sun Oct 4, 11:59pm, and Assignment 3 (optimize the agent) goes out the morning after, due Sun Oct 25 — A3 is still outside the API, so today's levers are not yours yet. They become yours in Assignment 4 (serve your own agent, out Nov 11), which is where block_size and gpu_memory_utilization stop being documentation and start being your numbers.

One thing to carry out of the room. Lecture 2 said the machine wants 295 and the memory permits 29, and it presented that as a hardware fact. Today two thirds of the gap turned out to be a data structure, and the binding constraint moved from the allocator to the SLO. Ask that question about every ceiling you are shown for the rest of the semester — including the ones in these notes.