Lecture 5 ended with an indictment: the 20-step designer agent submitted 97,000 prompt tokens of which only 7,700 were distinct, so 92% of everything the serving system prefilled, it had already prefilled. Today that 92% finally becomes recoverable machine time. The enabling fact is not an engineering trick but a property of the architecture — causal attention makes the KV cache for a token prefix a pure function of that prefix, so identical prefixes have identical caches and can be shared with no approximation at all. By the end you should be able to walk RadixAttention's tree operations by hand, compute what a given hit rate is worth in prefill FLOPs and TTFT on the agent ledger, say when chasing a cached prefix across engines beats load balancing, and argue LRU against cost-aware eviction with numbers — which is exactly the design a final project on this material would have to defend.

Where this sits

Lecture 5 measured the waste (92% re-sent tokens on the 20-step ledger; §13.5 turns that ledger into the ≈60× prefill:decode compute ratio) and the serving meetings of the last four weeks built the machinery: paged KV blocks that can be referenced rather than owned (batching and scheduling, Oct 7), prefill/decode separation (disaggregation, Oct 14), and a smaller per-token cache (KV-cache optimization, Oct 28). Today the pieces assemble into a cache: paging made KV blocks shareable, the radix tree decides what to share, the scheduler decides whether the sharing ever happens, and the eviction policy decides who pays when memory runs out. Monday (pruning and quantization, Nov 9) opens the third attack on the same bill — shrinking b, which among other things doubles how many prefixes fit before eviction matters at all.

Learning objectives

By the end of this class you should be able to:

1. Explain why identical token prefixes have identical KV caches under causal attention, and why one divergent token invalidates everything after it — a prefix property, not fuzzy matching.
2. Walk RadixAttention's match/split/insert/evict operations by hand on a request sequence, and say why eviction must be leaf-only.
3. Compute what a hit rate is worth: prefill FLOPs saved, total-task speedup via Amdahl's law at the 60:1 prefill:decode ratio, and per-step TTFT on the 20-step agent ledger.
4. Decide, with a millisecond budget, when prefix-affinity routing across engines beats load balancing — and when replicating a hot prefix beats both.
5. Decide fetch-versus-recompute for a cached prefix from its KV bytes, its recompute FLOPs, and a link bandwidth, and compute the break-even bandwidth.
6. Argue LRU against cost-aware eviction with numbers, design an admission rule that keeps singletons out, and state why the radix tree constrains both — a final project in one sentence.

13.1 The 92% bill, restated as a cache workload

Restate the ledger, because every number today hangs off it. Lecture 3's user-side session: 10 turns, 51,250 tokens in, 1,500 out, 84% of the input re-sent, ≈$0.18 with 74% of the bill going to re-sent prefix. Lecture 4's designer-side version of the same behavior: the 20-step agent under the keep-until-full context policy submits 97,000 prompt tokens of which 7,700 are distinct — 92% re-sends, because every step's prompt is the previous step's prompt plus a suffix. Lecture 5 called this an indictment of the workload. Today, convert it to machine time and it becomes an opportunity.

Name what a prefix cache stores, because it is not text. It stores the KV blocks the batching and scheduling meeting (Oct 7) taught the allocator to page: the session's 7,700 distinct tokens cost 7,700 × 512 KiB = 4.04 GB of KV under MHA (1.01 GB under GQA-8) — one agent session's entire reusable state is roughly two 4K-sequences' worth of the 62.5 GB budget from Lecture 2. The bytes are affordable; what was missing until today is the machinery to find and reuse them.

Every cache answers three questions, and they structure the lecture. Correctness — when is serving cached state instead of recomputing it sound? (§13.2: always, for exact prefixes, by theorem.) Lookup and placement — how do you find the longest reusable prefix, on one engine (§13.3–13.4) and across a fleet (§13.6–13.7)? Replacement — when memory runs out, who pays? (§13.8, and then you, from Nov 4.)

The punchline to carry into the papers: the agent loop's worst property as a request stream — it re-sends a growing transcript on every step — is its best property as a cache workload. The stream that looked pathological in Lecture 5 is the most cacheable traffic a serving system will ever see.

13.2 Why exact-prefix reuse is sound — and why it must be exact

The correctness argument is three sentences, and it is a theorem about the architecture rather than a heuristic. In a causal transformer, the K and V vectors at position i are linear projections of the hidden state at i, and that hidden state depends — through every attention layer — on the tokens at positions ≤ i and on nothing after. Therefore two requests whose first k tokens are identical have identical KV entries for those k positions, exactly. Serving those entries from a cache instead of recomputing them changes no logit, ever. (One honest aside: floating-point reduction order can differ across batch compositions, so bit-level wobble exists; the reuse is exact at the level the model's semantics are defined, which is why nobody qualifies it in practice.)

The same dependency structure gives the invalidation rule. Change one token at position j and the hidden state at j changes, hence K/V at j, hence — because every later position attends over j — K/V at every position after it. A 10,000-token prompt that differs from a cached one at token 3 shares 2 usable tokens. This is a prefix property: matching is all-or-nothing from the left, and there is no partial credit for similarity.

Two non-obvious consequences follow. First, prefix, not substring: K/V are computed with positional information (RoPE rotates each head by the absolute position index) and depend on all earlier tokens, so a matching 2,000-token span in the middle of two different documents has different KV in each — different history, different positions. The reusable unit is "same tokens from position 0," which is precisely the set of paths from the root of a trie over token sequences. The data structure of §13.3 is forced by the math, not chosen for convenience.

Second, no fuzzy matching. Reusing KV computed for a "similar" prefix silently changes the output distribution — it is an accuracy intervention wearing a systems costume. Everything in the serving machinery of the last four weeks (Oct 7–Oct 28) was exact-output-preserving, and prefix caching stays in that family; that is why it ships enabled by default in every major engine while approximate techniques ship behind flags.

Close the loop with Lecture 4's context-as-cache framing, which now pays off literally: the designer who keeps the transcript append-only is preserving the prefix property on purpose. The summarize-and-compact policy (Lecture 4's 75,100-token variant) edits the middle of the context — it saves 23% of submitted tokens and invalidates the cache from the edit point at every compaction. Token savings and cache reuse are in tension; §13.5 prices the trade, and seed 1 asks it.

13.3 RadixAttention: the KV cache as a radix tree

SGLang's runtime keeps every KV block it has computed — for finished requests as well as running ones — organized as a radix tree keyed by token sequences. A radix tree is a compressed trie: edges carry token runs rather than single tokens, so a long unbranched stretch (a 1,024-token system prompt nobody diverges inside) is one node, and the tree stays shallow. Each node maps its token run to a range of paged KV blocks (the paged blocks from batching and scheduling, Oct 7; paging is what makes a block referenceable by more than one request) and carries a reference count and a last-access time.

Three operations, all on request arrival or completion:

• Match. Walk from the root along the request's prompt tokens; the longest shared prefix wins. Matched blocks are referenced, not copied — the request's reference count pins them — and only the unmatched suffix is prefilled. Prefill work drops from the full prompt to the suffix.
• Split. If the walk diverges inside an edge — the request shares 700 tokens of a 1,200-token run — the node splits at the divergence point into a 700-token parent and a 500-token child, and the new request branches off the parent. No KV is recomputed; blocks are re-pointed.
• Insert. When a request finishes prefill and decode, the KV it produced extends the tree, so the next request sharing it hits. In an agent loop this means step k's transcript is sitting in the tree when step k+1 arrives — the cache is warmed by the workload's own structure.

Eviction is LRU over leaves only. An interior node is, by construction, a prefix of every cached descendant; evict a parent while keeping a child and the child's blocks are orphaned — they are only meaningful as a continuation of KV that no longer exists (§13.2's invalidation rule, enforced structurally). So eviction proceeds leaf-inward, and reference counts make in-flight nodes unevictable. A pleasant emergent property: hot shared prefixes — the system prompt, the agent scaffold — sit near the root with many descendants and are structurally the last thing eviction can reach.

root
└── system prompt + tool schemas          (1,024 tok — near-root, many descendants, hot)
    └── user goal                         (256 tok)
        └── step-1 thought + call + result  (1,200 tok)
            └── step-2 thought + call + result  (1,400 tok)
                ├── branch A: candidate + critique  (640 tok)   ← leaf, refcount 0
                ├── branch B: candidate + critique  (640 tok)   ← leaf, refcount 0
                └── step-3 (in flight)              (…, refcount 1 — pinned)

What RadixAttention is not: an API change. The engine sees raw token streams and discovers the sharing itself — call this API-transparent, and hold the term for §13.9, where Parrot occupies the opposite corner. The paper reports throughput gains that are multiples, not percentages, on structured multi-call workloads, and the gain scales with the shared-prefix fraction of the prompt; Lecture 5's Parrot numbers (four production apps re-sending 3% / 94% / 72% / 99% of prompt text) are the workload-side reason the multiples are large — and the 3% app is the warning that they are not universal.

13.4 Cache-aware scheduling: order the wave so hits happen before eviction

Under memory pressure, hit rate is not a property of the request set — it is a property of the arrival order, and the scheduler controls the order. A batch of queued requests is a wave, not a sequence of independent events, and the engine gets to choose the traversal.

SGLang's policy is exactly this: schedule the waiting queue by longest matched prefix, which is equivalent to a depth-first traversal of the radix tree — finish one subtree's requests while its blocks are resident, then move on. The caveat is the classic one: any policy that prioritizes by affinity can starve the request that shares nothing, and what guard to add is a genuine design question — hold it for the discussion seeds rather than pretending the paper closes it.

Agent fan-out is the killer app, because the workload generates the shareable wave itself. Take n = 8 branches — best-of-n sampling, or Lecture 4's parallel tool calls — off a 4,096-token parent, each adding a 256-token suffix:

The punchline that organizes the serving arc from Oct 7 to today: paging made KV blocks shareable, the radix tree decides what to share, and the scheduler decides whether the sharing ever happens. All three are load-bearing; remove any one and the 92% stays unrecovered.

13.5 What a hit rate is worth: the savings arithmetic

Now price it, on the ledger, honestly. Define the hit rate h as the fraction of submitted prompt tokens served from cache. Perfect reuse on the 20-step ledger prefills only distinct tokens:

A ceiling, not a promise: it assumes no eviction between steps, no divergence (append-only context), and suffix-only prefill at every step. Every real system sits below it, and how far below is a measurable question.

Prefill is not the whole task, so run Amdahl, and the ledger fixes the split. The 20 steps submit 97,000 prompt tokens and generate about 80 output tokens each, ≈1,600 tokens in all; prefill and decode both cost 2N FLOPs per token, so the ratio is 97,000 ÷ 1,600 ≈ 60:1 — a property of the workload, not of the model, since 2N cancels. Normalize decode to 1 unit, prefill to 60:

So "12.6× on prefill" is "10.6× on the task" — decode is untouched and becomes the new majority term. Read the table's shape, because it is the economics of any eviction policy: the first 50 points of hit rate buy 2×, and the last 12 points buy another 2.3×. Hit rate is convex in value. Two policies that differ by a few points near the ceiling differ by more than two mediocre policies differ in total, which is why eviction — the thing that decides those last points — is worth a semester's work.

Latency gets the same treatment, per step rather than per task:

Note the structure of what remains: the cache removes suffix-independent recompute; it does not remove attention over the prefix. The suffix's 1.0 TFLOP of cross-attention and the bandwidth to read 4,465 tokens × 512 KiB = 2.3 GB of cached KV are the irreducible cost of context — that is the KV-cache optimization bill (Oct 28), and quantizing the cache (Nov 9) is what shrinks it.

Two more consequences before leaving the ledger. Memory rent: the savings are paid for in HBM — 4.04 GB (MHA) or 1.01 GB (GQA-8) per session, held for the session's duration. Connect to Lecture 5's stalled-session fact (13B-class, 0.82 MB/token: a 16,384-token session holds 13.4 GB, ~6 sessions per 80 GB card): the prefix cache converts a stalled session from pure waste into an asset if the session resumes before eviction. An eviction policy is deciding which stalled sessions to bet on. The workload changes shape: with perfect reuse, prefill:decode compute falls from 60:1 to 0.104 PFLOP vs 0.0218 PFLOP ≈ 4.8:1. Caching turns the agent from an extreme prefill workload into a merely prefill-leaning one — which reshuffles the disaggregation sizing (Oct 14), and is a discussion seed.

13.6 Preble: the same tree across many engines

One engine's cache is 62.5 GB behind 29 concurrent MHA sequences (Lecture 2); a real service runs tens of engines, and the tree in §13.3 lives separately on each. Now routing decides the hit rate before any engine's scheduler gets a vote, and two forces pull in opposite directions. Prefix affinity: send the request to the engine that already holds its prefix — maximize hits. Load balance: send it to the least-loaded engine — minimize queueing. Pure affinity creates hot spots: one popular system prompt pins its entire tenant population to one GPU while the rest of the fleet idles. Pure load balancing scatters an agent session's steps across engines, and every step misses a cache that exists somewhere else. Both extremes are measurably bad, and Preble's workload study is the measurement.

Preble's resolution is hierarchical scheduling: a global scheduler holds a fleet-wide, radix-tree-shaped map of which engine caches which prefixes, and per-engine local schedulers keep doing §13.4's job. The routing decision weighs the recompute the cached engine would save against the extra queueing it would impose — long shared prefixes are worth chasing across the fleet, short ones are not, so requests are effectively split by how much recomputable prefix they carry. (We describe the mechanism and the cost model's shape; the paper's policy names, thresholds, and reported numbers are its own, and the reading guide asks you to pull them out.)

The cost model is one we can build ourselves from course numbers:

There is a third option the two-force framing hides: replicate. A hot prefix can be cached on several engines, spending HBM to relax the tension — 2.15 GB per 4K MHA copy (0.54 GB GQA-8). A copy pays off when its hit traffic on the second engine exceeds the eviction damage it does there, and there is no closed form for that — which is exactly why this is a scheduling paper rather than a formula, and why the single-engine version is already hard enough for a project.

The punchline: on one engine the prefix cache was a data structure; across engines it is a placement problem. Routing and load balancing (Oct 26) already generalized it — cache locality is just the sharpest instance of per-replica state.

13.7 Mooncake: the prefix store at cluster scale

Reading-only; not scheduled. If the Preble discussion runs short, the break-even derivation below is the five minutes to spend.

Mooncake — the serving platform behind a large production assistant — takes both of today's ideas to their limit: prefill and decode run on disaggregated pools (disaggregation, Oct 14), and the KV cache becomes a cluster-wide store, spilling from HBM to CPU DRAM and SSD across machines and moving over RDMA. The prefix cache stops being a per-GPU LRU and becomes a storage tier with a global namespace. Under overload it makes admission decisions — SLO-aware early rejection of requests it would fail anyway — which is a cache admission policy grown up into cluster admission. (Capacity and goodput figures are the paper's; we take only the architecture's shape.)

What the architecture makes decidable is fetch versus recompute, and it is one division:

Read the GQA row twice: it makes remote prefix caching viable on ordinary datacenter networks — one more systems payoff of a model-architecture decision, in the same family as Lecture 2's 29-vs-116 sequences.

Note the asymmetry against every classical storage cache you have studied: here a miss has a second recovery path — recompute — whose cost is known in advance and proportional to prefix length. That is why cost-aware policies (§13.8) have real leverage in this domain, and why the question at cluster scale stops being "what to evict" and becomes "what to keep where" — a cache hierarchy with recompute as the backstop tier.

13.8 Eviction and admission: the design space, and the project

Memory runs out; someone pays. Put two eviction candidates on the table and score them with course numbers:

LRU evicts (a) without looking at the right-hand columns. A cost-aware score — something shaped like p(reuse) × recompute_ms ÷ GB freed — sees per-byte recompute costs that are nearly flat: 294 ms ÷ 4.29 GB ≈ 69 ms/GB for (a), 14 ms ÷ 0.27 GB ≈ 53 ms/GB for (b). The structural reason: recompute cost and KV size are both ≈ linear in prefix length, so cost/size roughly cancels and LRU is a defensible baseline. The tie is broken by three second-order terms, and they are where a winning policy lives:

1. The quadratic attention share. Long prefixes cost more per token to recompute — 24% of (a)'s FLOPs are attention against 2% of (b)'s (294 ms for 8K is 2.28× the 129 ms for 4K, not 2×). Cost density rises with length; strict LRU never sees it.
2. The miss penalty is not one number. With a DRAM or remote tier (§13.7), (a)'s miss might cost a 2.4 ms fetch instead of a 294 ms recompute — per-candidate, depending on where its bytes also live. Eviction from HBM and eviction from existence are different decisions.
3. Reuse probability is wildly non-uniform. A live agent session's next step will re-present its prefix with near-certainty within seconds; a completed one-shot query never will. p(reuse) carries more signal than either cost term, and recency is only its crudest estimator.

Admission is the cheaper half of the policy. A prefix seen once by a completed request occupies bytes at p(reuse) ≈ 0 — don't cache singletons. Agent-scaffold prefixes announce themselves by reuse within milliseconds, so simple rules work: cache on second occurrence, or cache only ancestors of branch points in the tree. (Marconi, §13.9, is an entire paper about admission-by-expected-reuse, in a setting where admission is forced to matter.)

One constraint sits over all of it: the policy operates on a tree, not on independent objects. Eviction is leaf-only (§13.3), so "evict the long cold prefix" may mean evicting its whole subtree, and keeping a deep node pins its entire spine. Classical cache theory assumes independent objects; this cache's objects have ancestry. That twist is what makes the problem worth a semester's work.

As a final project. The project is announced today: 12% of the grade, one-page proposal Oct 28, report and repository Dec 8. This section is a project in a box. Replay a request trace, implement admission and eviction for a fixed-capacity prefix cache, and measure how much prefill your policy recovers against tree-aware LRU and a Belady-style offline oracle you cannot beat — the honest gap to the oracle being the interesting number. Today's §13.5 arithmetic is the scoring function's logic, today's table above is move one, and an agent-shaped trace means LRU is the baseline to beat, not the answer. The parked competition page has the metric and the anti-overfitting rules written out; it is not running this semester, but it is the right spec to build against.

13.9 What the prefix property does and does not survive: Parrot and Marconi

Reading-only; not scheduled.

Parrot returns from Lectures 4 and 5 as the other end of a design axis. SGLang is API-transparent: it serves today's traffic unmodified and must discover structure in raw token streams — paying tree-matching on every arrival and reacting to sharing only after it has seen it. Parrot is API-extended: the application declares its structure (Semantic Variables — templates with placeholders, plus dependency edges between calls), so sharing detection is free and exact, and the scheduler sees requests before they exist. Perfect information, at the price of rewriting every application against a new API. The workload data says why both corners are populated: Parrot's four profiled apps re-send 3% / 94% / 72% / 99% of their prompt text. For the 94–99% apps, transparent caching captures nearly everything declared structure would; for the 3% app, a transparent cache spends matching effort and HBM discovering that there is nothing to reuse — which is admission control's job to prevent. Transparency costs discovery; declaration costs adoption; the percentages decide which price is worth paying.

Marconi is the contrapositive of §13.2's theorem. Hybrid attention/SSM models — Mamba-style layers, sliding-window attention — do not keep per-token KV; they carry fixed-size recurrent state. State after token i is not addressable per token: you cannot take "the first k tokens' worth" of a state vector for arbitrary k, so reuse requires an exact hit on a checkpointed state, and checkpoints exist only where the system chose to save them (block boundaries). Token-granular radix matching degrades to exact-checkpoint matching; the tree gets coarser; and because a checkpoint is cheap to store but only exactly reusable, the leverage moves from eviction to admission — predict which prefixes will recur and checkpoint those, judiciously, rather than everything or nothing. Marconi's eviction scores candidates by expected FLOP savings rather than recency (the formula and results are the paper's; the mechanism is what matters here).

The symmetry closes the lecture. §13.2 proved reuse sound because attention is causal and its state is per-token; Marconi shows that changing the state's structure changes the cache's design. Architecture and serving co-evolve — GQA moved the fetch break-even 4×, SSMs coarsened the tree, and quantization (Nov 9) changes b under everything cached. None of these layers can be designed alone; that has been Part II's refrain, and the prefix cache is its cleanest exhibit.

Discussion seeds

1. Append-only vs compaction. Lecture 4's summarize-and-compact policy saves 23% of submitted tokens and breaks the prefix at every compaction. Using §13.5's speedup formula, at what hit rate does append-only-with-caching strictly dominate compaction-without-caching? What context policy would you design if you knew the cache existed?
2. Fairness. Longest-matched-prefix scheduling starves the request that shares nothing. Design a starvation guard and say what it costs in hit rate on the fan-out workload of §13.4.
3. Disaggregation, revisited. Perfect reuse moves the task from 60:1 to 4.8:1 prefill:decode. How should the disaggregation lecture's prefill/decode pool sizing (Oct 14) change — and does the prefix cache belong to the prefill pool, the decode pool, or neither?
4. Replicate or route? What runtime signal would you use to decide a prefix is hot enough to copy (2.15 GB per 4K MHA copy) rather than to chase (≤129 ms of queueing tolerance)?
5. Transparent vs declared. Would you rewrite your Assignment 2 agent against Parrot-style declarations for guaranteed hits, or trust the radix tree? What does your answer assume about who operates the serving stack?
6. Sharing across tenants. Two tenants submit byte-identical system prompts. Sharing their KV is sound by §13.2 — but a cache hit is observable through TTFT. Is a prefix cache a timing side channel, and what would you give up to close it?

Key takeaways

Numbers worth memorizing

Self-check

Exercises

1. Fan-out at the memory wall. n = 16 sampled branches off an 8,192-token parent prefix, 512-token suffix each, reference 7B. Compute prefill tokens and KV bytes with and without radix sharing, the sharing factor, and the GQA-8 bytes. Solution sketch: Without: 16 × 8,704 = 139,264 tokens; 139,264 × 512 KiB = 73.0 GB — this does not fit the 62.5 GB budget, so sharing is not an optimization here, it is feasibility. With: 8,192 + 16 · 512 = 16,384 tokens; 8.6 GB. Factor 8.5× on both. GQA-8: 18.3 GB → 2.15 GB.
2. The API bill with a cached-input discount. Lecture 3's 10-turn session (51,250 in / 1,500 out, 84% of input re-sent) at the illustrative $3/$15 per Mtok, with cached input billed at 10% of the input rate. Old bill, new bill, ratio, and the output tokens' share of each. Solution sketch: Baseline: 51,250 · $3/1e6 + 1,500 · $15/1e6 = $0.154 + $0.0225 ≈ $0.176. With cache: distinct 8,050 · $3/1e6 = $0.0242; cached 43,200 · $0.30/1e6 = $0.0130; output $0.0225; total ≈ $0.060 — 2.9× cheaper. Output's share rises from 13% to 37%: caching moves the bill toward the tokens the model actually generates.
3. Eviction scoring, with a trap. The cache must free ~4.3 GB. Candidates: (a) one 8,192-token prefix, p(reuse in the next hour) = 0.1; (b) sixteen 512-token prefixes, each hit once per second. Compute bytes freed and expected recompute cost per hour for each choice. Solution sketch: (a) frees 4.29 GB; expected cost 0.1 × 294 ms ≈ 29 ms/hr. (b) frees 16 × 0.27 = 4.29 GB; cost 16 × 3,600 × 14 ms ≈ 820 s/hr of prefill — about 23% of the GPU-hour, four orders of magnitude worse. LRU also picks (a) here (it is the cold one), so build the trap: let (b) go idle for 10 minutes and then resume its 1/s cadence — LRU evicts all of (b) and eats the 820 s/hr. LRU fails on regular-but-sparse reuse, which is exactly the pattern of an agent waiting on a human.
4. Fetch vs recompute under GQA. A 4,096-token GQA-8 prefix holds 0.54 GB of KV; recompute is still ≈129 ms (weight GEMMs dominate and do not shrink under GQA — only the bytes do). Break-even bandwidth, and verdicts at 900, 50, and 12.5 GB/s. Solution sketch: Break-even = 0.54 ÷ 0.129 ≈ 4.2 GB/s. At 900 GB/s: 0.6 ms — fetch wins ≈215×. At 50 GB/s: 10.7 ms — fetch wins 12×. At 12.5 GB/s: 43 ms — fetch still wins 3×, where the MHA version (172 ms vs 129 ms) loses. GQA roughly quadruples the set of networks over which a remote prefix store beats recompute.
5. What better eviction can buy. On the 97,000/7,700 ledger, compute total-task speedup at h = 0.3, 0.6, 0.9, and the marginal value of one more point of hit rate at each. Solution sketch: speedup = 61 ÷ (60(1−h)+1): h = 0.3 → 1.42×; 0.6 → 2.44×; 0.9 → 8.71×. Marginal, per point: ≈0.02× at 0.3, ≈0.06× at 0.6, ≈0.8× at 0.9. The last points are worth ~40× the first — the margin lives in retaining the near-ceiling prefixes, not in gross hit rate.

Reading guide

Required — SGLang / RadixAttention. Read the RadixAttention section and its tree figures carefully: you should be able to redraw the tree after a given request sequence and state the eviction rule (LRU, leaves only) and the scheduling rule (longest matched prefix first) from memory. Read the frontend/language sections lightly — they matter as evidence that LLM programs have shareable structure, not for the DSL — and skip the compressed-FSM constrained-decoding material entirely; it is a different problem sharing a paper. Hold this question: the runtime never asked the application anything — what did that transparency cost, and where in the paper do you see the runtime working to rediscover structure that Parrot would simply have been told?

Required — Preble. Read the workload study first — the measured prefix-sharing fractions across workloads are the motivation; treat the percentages as their data rather than facts to memorize. Then the hierarchical design: global scheduler with a fleet-wide prefix map, local per-engine schedulers, and the cost model trading recompute savings against load. Skim the implementation. Hold this question: find the experiment where pure prefix affinity loses — what property of that workload makes load imbalance more expensive than cache misses?

Optional — Parrot. Re-read the Semantic Variable section and the application characterization table with Lecture 4 in hand; skip the DAG-scheduling machinery Lecture 5 already covered. Question: which of today's radix-tree mechanisms becomes unnecessary when structure is declared instead of discovered?

Optional — Mooncake. Read the architecture figure and the KV-cache-store and transfer sections; skim the overload/admission material for its shape (SLO-aware early rejection); skip cluster-operations detail. Question: at your cluster's actual link bandwidths, which of your prefixes clear the ≈4–17 GB/s fetch-vs-recompute bar?

Optional — Marconi. Read the sections explaining why SSM/hybrid state breaks token-granular reuse, then the admission and eviction policy; skip the architecture background if Lecture 2 is fresh. Question: which assumption of §13.2 failed — per-token addressable state — and what replaced exact-prefix matching?

Looking ahead

Monday (pruning and quantization, Nov 9) opens the third attack on the KV bill: prefix caching removed redundant recompute, and quantization shrinks b — for the weights and for every cached byte, so a 2× on b doubles how many prefixes fit before your eviction policy is ever consulted. Assignment 3 (optimize the agent) was due Oct 25 and Assignment 4 (serve your own agent) was due Nov 10, with Assignment 5 (optimize the full stack) due Dec 2 — prefix-cache hit rate is one of the levers A5 hands you, and the first one A3 could not reach from outside the API. Routing and load balancing (Oct 26) already generalized §13.6: cache locality as a per-replica property. Nov 18 and Nov 23 are the two classes on agent serving (notes 18–22) — the request streams that make today's caches hit or thrash, closing the loop opened in Lectures 4–5; the Nov 23 meeting on tool stalls and session state is where today's eviction question returns as a storage-tier question. And the final project was announced on Oct 26 (proposal Oct 28, report Dec 8, 12% of the grade). Lecture 5 measured the waste; the serving meetings of the last four weeks (Oct 7–Oct 28) built the pages; today reused them. If you want the eviction policy to be yours to write, §13.8 is your proposal already half-drafted — and the trace will not be kind to LRU.