The course's two halves meet today. Lecture 5 characterized the agent request stream — shared prefixes, short dependent decodes, inter-request dependence, branch variance, stalls — and ended on the indictment that the completions API throws all five away: the server receives finished strings, one at a time, from a program it cannot see. Part II then built engines that claw fragments back implicitly: RadixAttention rediscovers prefixes token by token at runtime, Llumnix migrates load it could not predict, chunked prefill defends decodes against prompts it did not know were coming. Today we change the API instead of the engine and ask what becomes possible when the application tells the system its structure. Parrot's answer is Semantic Variables — string placeholders that make the request DAG explicit — and it buys exactly three things, each of which we will price on the Lecture 5 ledger: dependent requests execute server-side, the scheduler deduces what actually matters, and prefix detection becomes a structural fact instead of a runtime guess. By the end you should be able to schedule the 20-step designer task with and without structural information and account for every second of the difference — and recognize that everything you are doing client-side in Assignment 5, due Dec 2, is what these systems attempt from the other side of the API. Today opens a two-meeting block on agent serving; this is its interface layer.

Where this sits

Every meeting from GPU programming (Sep 30) through speculative decoding (Nov 11) built the serving stack for requests that arrive independent and opaque: kernels, memory, batching, paged caches, prefix reuse, quantization, speculative decoding, and — three weeks ago, on Oct 26 — routing and load balancing across replicas. Lecture 5 predicted that stack would leave money on the table for agents, and quantified how much. Today the two threads finally touch: Parrot keeps the engine and changes the interface, declaring the structure Lecture 5 showed the API destroys. The rest of the block takes the structure declared today and follows it down the resource stack: Monday, Nov 23 goes into execution, where tool calls stall sessions and sandboxes hold the other half of the state (Lecture 19); into storage, where a session's KV outlives every request that touches it (Lecture 20); and into composition, where one agent becomes many (Lecture 21) — while the scheduler that must serve all four at once, including the dynamic agentic programs no declared DAG can describe, is this class's own second half (Lecture 22).

Instructor notes — Timing plan

75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221).

Time	Segment	Notes
0–5	Framing; A5 in flight; the block ahead	One line of admin, then: "Part I told you what the API destroys. Today: what declaring it buys — and how this class's second half and Monday spend the declaration."
5–15	§18.1 Five properties, two columns	The recap table plus the implicit-recovery column. End on "a guess about the past vs a fact about the future."
15–23	§18.2 Semantic Variables	Mechanism only. Board the 16-map DAG; it gets reused twice.
23–31	§18.3 Payoff 1: round-trips	Derive 4.75 s live; flag both assumptions loudly.
31–44	§18.4 Payoff 2: batch vs expedite	The 21.6 s vs 6.4 s pair. Protect the closing sentence: per-request TPOT got worse while the user got faster.
44–48	§18.5 Payoff 3, compressed	In class: the 1/(1−r) formula and the four values. The ledger derivation is on the page.
48–63	§18.6 Centerpiece — three configs	Never cut. The A/B/C table and the lever decomposition must be on the board when the bell goes.
63–73	§18.7 §6 limits and the design axis	Draw the axis. InferCept gets the mechanism, not the arithmetic.
73–75	§18.8 in one sentence + close	"Everything you are doing in A5 client-side is what these papers do server-side." Point at Assignment 5.

Reading-only, not scheduled: §18.8 in full (one sentence at minute 74; the mapping table is assigned reading), the ledger derivation in §18.5, and the InferCept recompute-vs-swap arithmetic in §18.7 — if five minutes appear, do the 0.86 s vs 0.42 s comparison live; otherwise assign it with exercise 4.

If running long: compress §18.3 to the formula dead time = E · (RTT + Q) and the 4.75 s result; cut §18.5 to the four ceilings. Never cut §18.6 — without the decomposition table the lecture is three disconnected tricks instead of one argument.

Learning objectives

By the end of this class you should be able to:

1. Name the five properties of the agent request stream, and for each say what Part II's engines recover implicitly and what only an API change can provide.
2. Explain what a Semantic Variable is, what crosses the wire when an application uses one, and why the server then holds the request DAG before any request runs.
3. Compute the dead time a client-side dependency chain pays in round-trips, and what server-side dataflow execution recovers.
4. Given a DAG, decide which requests to batch for throughput and which to expedite for latency, and quantify the gap between per-request metrics and the user-visible end-to-end latency.
5. Convert a prompt-repeat rate r into the prefix-reuse compute ceiling 1/(1−r), and explain why retention policy dominates detection cleverness at high r.
6. Place SGLang, Parrot, InferCept, and Autellix on the axis "how much of the program crosses the API," and say which parts of an agentic workload each point serves best.

18.1 The information the API destroys, and what Part II recovered without it

Start from the ledger. Lecture 5 instrumented the 20-step designer task and found five properties, each with a number attached. Beside each, write what Part II built to recover it — without ever being told it was there.

Property (Lecture 5)	Canonical number	Implicit recovery (Part II)	What stays lost
Shared prefixes	97,000 tokens submitted vs 7,700 distinct = 92% re-sent; reuse ceiling 97,000 ÷ 7,700 ≈ 12.6×	RadixAttention: match each prompt against a radix tree of cached KV	Whether the prefix will return; when to evict
Short dependent decodes	prefill ≈ 60× decode compute → 97,000 ÷ 60 ≈ 1,617 ≈ 1,600 output tokens, ~80/step	Chunked prefill absorbs the prompt spike; continuous batching keeps decodes fed	Which decodes are latency-critical
Inter-request dependence	request k+1's prompt contains request k's output — 19 edges in 20 steps	Nothing. The server sees independent requests	The DAG itself
Branch variance	fan-outs and retries make per-request length unpredictable	Llumnix migrates load after the imbalance appears (routing and load balancing, Oct 26)	The fan-out's existence, ahead of time
Stalls	13B-class session at 16,384 tokens holds 0.82 MB/token × 16,384 = 13.4 GB; ~6 per 80 GB card	Eviction heuristics, swap-on-pressure	How long the stall will last; whether the session returns

Read the third column honestly: it is impressive engineering, and every piece of it is a guess about the past. RadixAttention discovers that this prompt shares 5,000 tokens with an earlier one — after paying O(prompt length) matching, and without knowing whether the shared region returns. Llumnix rebalances a replica after a fan-out landed on it. Chunked prefill defends decodes from a prompt it learned about upon arrival. Nothing in the column recovers dependence, and nothing distinguishes an intermediate step from a final answer, because neither is present in any request — they are properties of the client program, and the program never crosses the wire.

Declared structure is different in kind, not degree: it is a fact about the future. A guess costs runtime work and is sometimes wrong; a fact is free at request time and admits scheduling that guessing never can. You cannot expedite "the last request of the task" if you do not know which request is last.

That framing gives today's design axis, which we resolve in §18.7: how much of the program crosses the API? SGLang answers nothing — recover everything at runtime inside the engine. Parrot answers templates plus the DAG. Autellix — today's second half — answers enough to schedule the whole agentic program. Today is the middle point.

Instructor notes

Minutes: 10. Board: The five-row table, but write the third column second — ask the room to fill it from Part II before revealing. Then write "guess about the past / fact about the future" under it. Ask the room: "Which of the five properties has no implicit recovery at all?" Dependence — and have them say why: it lives in client code the server never receives. Expect confusion: Students think RadixAttention "knows" about prefixes. It observes overlap; it does not know the prefix returns 19 more times, which is why its eviction can be wrong.

18.2 Semantic Variables: the mechanism

What Parrot moves across the wire is small and precise. An application registers semantic functions: prompt templates with named string placeholders for inputs and outputs. Submitting a call passes variable references, not finished strings. When the output variable of one call appears as an input of another, that is an edge — and because the references arrive before the values exist, the server holds the request DAG before any request runs.

Be clear about what this is not. It is not a new model interface and it is not speculation: the strings the model eventually sees are the same strings the client loop would have assembled. Only the labels on their parts are new. Where the classic API submits one opaque byte string, Parrot's submission distinguishes the constant region (the template, shared across every call to the same function) from the variable regions (the placeholders), and declares the identity of a value across requests — "this input is that output" — instead of leaving the server to infer identity from token overlap.

Board the running example: a map-reduce summarizer in the shape Lecture 4's control-flow section sketched and Parrot's own evaluation uses. Sixteen summarize_chapter(doc_k) → summary_k calls, one reduce(summary_1 … summary_16) → answer. As semantic functions: one template registered once, sixteen calls that differ only in one input variable, sixteen output variables all flowing into a single sink. The server sees sixteen edges converging on one node — a picture, where the classic API sees seventeen unrelated strings arriving in some order.

Three payoffs follow from holding that picture, and they are the next three sections:

1. The dataflow executes server-side — no client round-trip per edge (§18.3).
2. The scheduler deduces objectives from DAG position — batch the interior, expedite the sink (§18.4).
3. Prefix identity is structural — a template id comparison, not runtime token matching (§18.5).

One honest note now, expanded in §18.7: the DAG must be expressible as templates and static edges. Parrot's §6 concedes the rest, and "the rest" is exactly the loop you wrote in Assignment 2. Hold that.

Instructor notes

Minutes: 8. Board: The 16-map / 1-reduce DAG, large — it gets annotated in §18.4 and contrasted in §18.6. Under it: "constant region declared, variable regions declared, edges declared." Ask the room: "What information did the client have to give up to get this?" Nothing it wasn't already computing — the loop already knew its own dataflow; the API just had nowhere to put it. Expect confusion: "Semantic" suggests the server understands meaning. It understands plumbing: which string goes where. No model involvement in the DAG at all.

18.3 Payoff 1 — the dataflow executes server-side: pricing the round-trips

Under the classic API, every edge in the DAG is a detour through the client: the server finishes request k, ships the output over the WAN, the client parses it, splices it into prompt k+1, ships that back, and the new request re-enters the queue as a stranger.

The subtler cost is the stall property riding along. Between response k and request k+1, the session's KV either sits in HBM — and Lecture 5's 13.4 GB session is one of only ~6 that fit on an 80 GB card, so each stalled second idles roughly a sixth of the card's KV capacity — or it is evicted and re-prefilled on return. Either way the client loop converts every edge into a small stall, 19 times.

With the DAG on the server, an edge costs nothing: the moment the parent's output variable fills, the child's prompt is materialized in place and the child enters the queue — same machine, same scheduler tick. Better, because the edge is visible before the parent finishes, the opportunity exists to prefill the child's constant regions early and overlap work across the edge; treat the exact overlap granularity as an opportunity the visible edge creates rather than a guarantee of any implementation. For pure LLM→LLM edges the stall window collapses to zero. Tool-call edges still stall — the server cannot run your Python — and that residue is InferCept's problem, previewed in §18.7 and taken up in full on Monday, Nov 23.

The punchline, sized: on our chain, payoff 1 alone is worth ≈4.75 s of a task we will shortly price at ≈43.6 s end-to-end — about 11%. Real, linear, and not the headline. The headline is next.

Instructor notes

Minutes: 8. Board: The formula, then 19 × 250 ms = 4.75 s. Write "ASSUMED" over both the 50 and the 200. Ask the room: "Which term do you control by moving your client into the same datacenter?" RTT only — Q is the server's queue, and declared structure is what removes it (the child is expected, not a stranger). Expect confusion: Students assume HTTP keep-alive or streaming removes this. The bytes still cross twice per edge and the request still re-enters admission; only moving the loop removes the loop's cost.

18.4 Payoff 2 — deduced scheduling objectives: batch the interior, expedite the sink

Batching and scheduling (Oct 7 and Oct 14) built schedulers around per-request service objectives — TTFT and TPOT for each request — because per-request is all the API exposes: every request might be a human staring at a stream, so every request is treated as latency-critical. The DAG collapses that fiction. Only the sink's completion is user-visible; an intermediate's TPOT is worth nothing to anyone. The correct objective is per-application: throughput-schedule the interior, latency-schedule the sink.

Work it on the board DAG. Sixteen maps, each a 6,000-token prompt producing a 250-token summary; one reduce over the sixteen summaries plus ~200 instruction tokens, producing a 400-token answer. Per-map KV at GQA-8: mean residency ≈ 6,125 tokens × 128 KiB ≈ 0.80 GB — all sixteen fit in the 62.5 GB budget with room to spare (16 × 0.80 = 12.8 GB used of 62.5).

The reduce is identical under both policies — it is the sink, so both expedite it: prefill 16 × 250 + 200 ≈ 4,200 tokens → 56.6 TFLOP → 0.114 s; decode 400 tokens at (13.5 + 0.58 GB KV) ÷ 3,350 ≈ 4.2 ms → 1.68 s; call it 1.8 s.

End-to-end: 21.6 s versus 6.4 s — 3.4×. And now read the per-request metrics: every individual map's TPOT got 1.8× worse, 4.3 → 7.85 ms. The per-request metric and the user-visible metric moved in opposite directions. That sentence is the section.

Name the failure mode precisely, because it will appear in production dashboards near you: a goodput-maximizing scheduler in Lecture 2's sense would proudly report sixteen maps each meeting a 10 ms TPOT SLO — and a user who waited three times longer than necessary. Goodput against per-request SLOs on intermediates measures compliance with a promise nobody needed.

Who pays for expediting sinks? The interiors of other applications. A near-batch-1 stream burns capacity (Lecture 2: 0.34% arithmetic utilization), and Parrot's scheduler can fund it precisely because the DAG tells it which requests tolerate batching — the interiors absorb what the sinks spend. Without the DAG, every request must be treated as possibly-latency-critical, and the subsidy has no source.

Instructor notes

Minutes: 13. Board: Annotate the §18.2 DAG: "interior → batch" on the sixteen maps, "sink → expedite" on the reduce. Derive 7.85 ms live. Then write the pair "TPOT 4.3 → 7.85 ms (worse)" and "E2E 21.6 → 6.4 s (3.4× better)" side by side and box both. Ask the room: "Your dashboard shows every request meeting its TPOT SLO. What question has it not answered?" Whether any application finished sooner — the dashboard has no column for the DAG. Expect confusion: The serialized 19.8 s looks like a strawman. It is the honest consequence of "every request is latency-critical" under contention: expedited capacity is scarce by definition, so universal criticality means queueing for it. The alternative — batch everything — is Config A of §18.6, and it loses on the sink instead.

18.5 Payoff 3 — prefix detection becomes structural

Runtime recovery first, to be fair to it. RadixAttention keeps cached KV in a radix tree keyed by token ids; each arriving prompt walks the tree, longest-match. Correctness of reuse is guaranteed — matched tokens are genuinely identical. But the work is O(prompt length) per request — over the 20-step task, matching against 97,000 submitted tokens in aggregate — and retention is a guess: LRU does not know this prefix returns 19 more times, and one wrong eviction re-prefills the whole shared region.

With declared structure, both halves change. The template's constant region is shared by declaration: detection is a pointer comparison on the semantic-function id — O(1) per request, independent of prompt length. And retention becomes informed: the DAG says this prefix has k unscheduled descendants, so pin it until k reaches zero. Eviction with a reference count, not a heuristic.

How much is at stake is an application property. Parrot profiles four production-style applications and measures the fraction of prompt text that repeats across requests: 3% / 94% / 72% / 99%. Convert repeat rate r into the ceiling on prefill-compute reduction:

Two lessons in those four numbers. First, the win is enormous and wildly app-dependent: the 3% app — per-request-unique prompts — gets nothing, which is why structure declaration must be optional rather than a tax on every client. Second, at r = 99%, a single runtime cache miss costs more than all remaining matching overhead combined — one eviction re-prefills the 99% region — so retention policy dominates detection cleverness exactly where the stakes are highest. Exercise 3 quantifies this: at a 20% miss rate the 100× ceiling collapses to under 5×.

On our own ledger (reading — in class this section is the formula and the four values):

Be honest about the delta: RadixAttention captures nearly the same 2.43 s whenever nothing is evicted. Payoff 3's advantage over runtime recovery is not the hit case — it is the miss cases it prevents (pinning by reference count) and the matching work it deletes. Structure wins at the margins, and at r = 99% the margins are the game.

Instructor notes

Minutes: 4 (compressed — formula and four values; the ledger derivation is assigned). Board: 1/(1−r) and the four ceilings in a row. Circle 100× and write "one eviction kills it." Ask the room: "Which app should decline to declare structure?" The 3% one — and the follow-up: what does that imply about making declaration mandatory? Expect confusion: Students equate payoff 3 with "prefix caching, which we already have." The cache is the same; what changes is who decides retention — a reference count that knows the future versus an LRU that guesses.

18.6 Scheduling the Lecture 5 graph with and without structural information

The centerpiece. One task — the 20-step, 97,000-token designer chain — three serving configurations, one table, every cell derived from numbers already on the board.

Config A — vanilla engine, client loop, no prefix cache. Prefill all 97,000 submitted tokens: 2.64 s. Decode 1,600 tokens in the ambient shared batch — the server has no reason to treat this session differently from anyone's chat: 1,600 × 22.6 ms = 36.2 s. Round-trips: 19 × 250 ms = 4.75 s. End-to-end ≈ 43.6 s.

Config B — RadixAttention engine, client loop (Part II's implicit recovery). Prefill falls to the distinct tokens, 7,700 → 0.21 s — generously assuming every lookup hits and nothing is ever evicted. Decode: unchanged at 36.2 s, because there is no basis to expedite — every request in the stream looks like anyone's request. Round-trips: unchanged at 4.75 s — the loop still lives on the client. End-to-end ≈ 41.2 s. Implicit recovery bought 2.4 s of 43.6 — about 6%.

Config C — declared structure, Parrot-style. Prefill 0.21 s, now structural and pinned — no miss risk. Round-trips: 0 — the chain executes server-side. Decode: the DAG shows a pure chain, so every decode is on the critical path; there are no batchable intermediates, and the deduced objective is to expedite the whole chain: 1,600 × 4.3 ms = 6.9 s. End-to-end ≈ 7.1 s.

	Prefill	Decode	Round-trips	End-to-end	vs A
A — vanilla, client loop	2.64 s	36.2 s	4.75 s	43.6 s	—
B — RadixAttention, client loop	0.21 s	36.2 s	4.75 s	41.2 s	1.06×
C — declared structure	0.21 s	6.9 s	0	7.1 s	6.1× (5.8× vs B)

Now decompose the 36.5 s between A and C by lever, because the decomposition is the argument:

Lever	What it needs	Seconds saved
Prefix reuse (2.64 → 0.21)	runtime recovery or structure	2.43
Round-trip removal (4.75 → 0)	structure — server-side dataflow	4.75
Objective-aware decode (36.2 → 6.9)	structure — knowing the chain is the critical path	29.3

Read it aloud. On a chain, the dominant lever — 29.3 of 36.5 s — is the one only structure can pull: knowing these decodes are latency-critical. On the fan-out DAG of §18.4 the dominant lever flipped to batching the interior. The five Lecture 5 properties buy different amounts depending on DAG shape, and a serving system for agents needs the shape to know which lever to pull. That is the lecture in one sentence.

The honesty paragraph, which is part of the result and not a disclaimer. Config C's 4.3 ms decode sits near the batch-1 floor, where Lecture 2 priced utilization at 0.34% — it costs the server the throughput of every slot it vacates and is fundable only when other tenants' interiors absorb the batching (§18.4's subsidy). The 250 ms per edge is an assumption: a fast client on the same coast shrinks the round-trip lever toward 19 × 50 ms ≈ 0.95 s, while a chatty tool-using client grows it. The 6.1× headline is scenario arithmetic, not a benchmark. Parrot's own evaluation reports the same shape — up to an order of magnitude end-to-end on DAG-rich applications, and little on chat-like low-repeat workloads; take the shape from the paper and the mechanism from this table, and treat any single multiplier, theirs or ours, as a function of the scenario it was computed in.

Instructor notes

Minutes: 15. Never cut. Board: The three-config table built column by column — every cell derived, none announced. Then the lever table beside it. Circle 29.3 and write "chain: the lever only structure can pull." Ask the room: Before revealing Config B's decode column: "RadixAttention just saved 92% of prefill. What does it do about the 36.2 s of decode?" Nothing — and make them articulate why: nothing in any request says these decodes are special. Expect confusion: Two, reliably. (1) "Why can't B expedite too?" It could expedite everyone, which is Config C's cost without its information — the server cannot fund universal expediting. (2) "Is 6.1× Parrot's number?" No — it is ours, under stated assumptions; the paper's numbers are its own workloads' numbers. Same shape, different scenario. If short on time: Drop the honesty paragraph's second half (the RTT sensitivity) and point at exercise 5; never drop the lever decomposition.

18.7 What the structure cannot say: §6, and the design axis

Parrot's §6 concedes two things, and the concession is load-bearing. Dynamic control flow: when the next request depends on inspecting the previous output — a while not done loop, a branch on the model's own judgment — the edge cannot be declared, because it does not exist until the value does. Native functions: arbitrary client code between calls — your parser, your retry logic, your tools — cannot ship to the server. Now look at what you built in Assignment 2: assemble → invoke → parse → tool → repeat, with termination decided by the model's own output. That loop is exactly the program Parrot cannot see. Its DAG is generated one edge at a time, at runtime, by the very outputs the server would need in advance.

So the agent workload splits. Template-shaped substructures — map-reduce, fixed pipelines, multi-agent fan-outs — declare cleanly and execute server-side. The outer adaptive loop stays on the client, still round-tripping, still stalling. Size the damage honestly against §18.6: if every edge of the 20-step task is a tool-call edge in the Assignment 2 style, payoff 1 evaporates entirely — the loop must come home for every tool — and only payoffs 2 and 3 partially survive (the templates inside each step still declare their constant regions; the steps' decodes can still be marked critical if the session is declared even when its length is not).

Draw the axis on the board, left to right by how much crosses the API:

• SGLang — nothing crosses. Runtime recovery inside one engine; works for every client unmodified; can never see the future.
• Parrot — templates and the static DAG cross. Sees the declared future; blind to dynamic control flow and native functions.
• Autellix (Lecture 22) — treats the whole agentic program as the schedulable unit, using program-level statistics rather than shipping code or a static DAG to the server; how it schedules what it cannot be told is where this class's second half begins.

InferCept sits orthogonal to the axis: it attacks the stall property directly and needs no DAG at all. When a request intercepts — a tool call, a human turn — the engine chooses per session among three fates for the KV, by modeled cost. On Lecture 5's stalled session (reading — assign with exercise 4 if time is short):

The punchline: implicit recovery, declared structure, and program-aware scheduling are not competitors on one benchmark. They are points on "how much does the application tell the system," and the agent workload genuinely contains parts best served at each point — which is why all three papers coexist rather than one having won.

Instructor notes

Minutes: 10. Board: The axis, three labeled points, and under Parrot's point: "§6: dynamic control flow, native functions — the Assignment 2 loop." Leave the axis up; the whole block hangs off it, and today's second half (Lecture 22) starts from it. Ask the room: "Your A1 agent calls a tool every step. Which of today's three payoffs survives?" Not payoff 1; payoffs 2–3 partially. Make them defend "partially." Expect confusion: Students hear §6 as a flaw. It is a scope statement — the honest kind — and the reason the rest of this block exists. Papers that state what they cannot do are telling you where the next paper is.

18.8 Closing symmetry: Assignment 5 is the client-side mirror

(Reading section — in class this is one sentence at minute 74.)

Assignment 5, out October 26 and due December 2, asks you to optimize an agentic system from the client seat. Every lever you pulled has a server-side twin in today's material, and the mapping is worth writing down before your A5 numbers are final:

Your A5 lever (client)	Today's twin (server)
Context policy — Lecture 4's 97,000 / 83,500 / 75,100 tokens for keep-until-full / pinned window / summarize-and-compact	Prefix retention: what the cache pins is the mirror of what your context keeps
Capping tool output — Lecture 4's uncapped 20,000-token result at step 4 of 20 → 320,000 prompt tokens	No twin. Once the tokens are in the declared prompt, no server-side system can un-submit them. Only the client can decline to say something
Your measured re-send rate	The 3 / 94 / 72 / 99% that decides whether structure declaration pays at all
The dead time you clocked between steps	Payoff 1 — the 4.75 s the server-side dataflow deletes
End-to-end wall clock vs per-call latency in your report	Payoff 2 — the objective a per-request dashboard cannot see

The numbers you measured are the opportunity these papers chase: 92% re-sends is the prefix payoff, the inter-step dead time is the round-trip payoff, the gap between your wall clock and your per-call latencies is the objective payoff. If your A5 measurements disagree with the ledger's — they will, somewhere — the disagreement is a finding, not an error.

The division of labor going forward: the client optimizes what only it can know (what deserves context, what a tool may return); the server optimizes what only it can see (the batch, the cache, the other tenants). The API between them is where this course's two halves meet — and Assignment 5, the full-stack optimization pass on the system you serve yourself, sits exactly on that line.

Instructor notes

Minutes: 1 — one sentence: "everything you are doing in A5 client-side is what these systems attempt server-side; the mapping table is in the notes, read it tonight while your numbers are still moving." Board: Nothing new. Expect confusion: None — but A5 is three weeks out, so spend the attention pointing at Assignment 5: same system, same levers, with the serving stack in scope.

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Client-loop dead time, 20-step chain	19 × 250 ms = 4.75 s	E · (RTT + Q), illustrative 50 + 200 ms
Fan-out: per-request vs DAG-aware	21.6 s vs 6.4 s ≈ 3.4×	§18.4, maps batched at 7.85 ms TPOT
Per-map TPOT under DAG-aware batching	4.3 → 7.85 ms (worse, and correct)	(13.5 + 16 × 0.80) GB ÷ 3,350 GB/s
Prompt-repeat rates, Parrot's four apps	3 / 94 / 72 / 99%	ceilings 1/(1−r): 1.03 / 16.7 / 3.6 / 100×
20-step prefill, no reuse vs perfect reuse	2.64 s vs 0.21 s (Δ 2.43 s)	97,000 vs 7,700 tokens × 13.48 GFLOP ÷ 494.5 TFLOP/s
Three configs, end-to-end	A 43.6 / B 41.2 / C 7.1 s → 6.1×	§18.6, assumptions as stated
Chain lever decomposition	prefix 2.43, round-trips 4.75, decode objective 29.3 s	§18.6
Stalled 13.4 GB session: keep / recompute / swap	~1/6 card per s / 0.86 s / 0.42 s	§18.7, InferCept's three-way choice

Self-check

Exercises

1. Round-trip budget. A 50-step chain agent, one dependence edge per step boundary, RTT 80 ms, server re-admission queueing 150 ms. (a) Dead time under the client loop? (b) If the task's server-side compute is 12 s, what fraction of end-to-end is dead time, and what does server-side dataflow execution recover? Solution sketch: (a) 49 × 230 ms = 11.27 s. (b) E2E ≈ 23.3 s, dead ≈ 48%; removing it is a 1.94× speedup from payoff 1 alone. Linear in edges — long chains are where payoff 1 stops being a rounding error.
2. Fan-out at the memory ceiling. Redo §18.4 with 64 chapters (same 6,000-in / 250-out maps, 0.80 GB KV per sequence, GQA-8, 62.5 GB KV budget). (a) Do all 64 fit? (b) Batched-map decode TPOT and time? (c) Speedup over per-request-scheduled maps. Solution sketch: (a) 64 × 0.80 = 51.2 GB < 62.5 → fits. (b) Bytes/step = 13.5 + 51.2 = 64.7 GB → 19.3 ms TPOT; 250 tokens → 4.83 s; prefills 64 × 0.164 = 10.5 s → maps done in 15.3 s batched. (c) Sequential: 64 × 1.24 = 79.4 s → 5.2×. Note what changed: prefill now dominates the batched path (10.5 of 15.3 s) — the next lever is prefix sharing across the maps' common instruction template, plus chunked-prefill interleaving. Levers compose.
3. Repeat-rate portfolio. For the four profiled apps (r = 3%, 94%, 72%, 99%), compute the prefix-reuse compute ceiling 1/(1−r), then the realized reduction if an LRU cache misses the shared prefix on 20% of requests (treat a miss as full re-prefill of the repeated region). Solution sketch: Ceilings 1.03 / 16.7 / 3.6 / 100×. With 20% misses, repeated-region compute per request is 0.2r + (1−r) of baseline, so the realized factor is 1/(1−0.8r): 1.025 / 4.03 / 2.36 / 4.81×. The r = 99% app falls from 100× to 4.8× — eviction policy, not detection, is worth ~20× there, and structural pinning (reference counts from the DAG) is the fix.
4. Recompute-vs-swap break-even. Using self-check 6's numbers, find the PCIe bandwidth below which discard-and-recompute beats swapping for the 13.4 GB session, and determine whether a context length exists below which recompute wins at 64 GB/s. Solution sketch: Swap time = 2 × 13.4 ÷ BW; set equal to 0.86 s → BW ≈ 31 GB/s — below that, recompute wins. Per token: swap = 2 × 0.82 MB ÷ 64 GB/s = 2.56e-5 s; recompute = 26e9 ÷ 494.5e12 = 5.26e-5 s. Both are linear in context length S, so their ratio (~2.1× in swap's favor) is independent of S — recompute never wins on time at these figures, at any length. The real trade is that swap consumes PCIe while recompute consumes GPU FLOPs — different resources — so the answer in a live system depends on which one is contended, not on S.
5. The 6.1× audited. Recompute §18.6 when (a) RTT is 10 ms and requeue 50 ms (fast client, quiet server); (b) the ambient TPOT is already 8 ms because the server runs at moderate batch; (c) both. Which lever survives every scenario? Solution sketch: (a) A = 2.64 + 36.2 + 19 × 0.06 = 40.0 s; C = 7.1 s → 5.6×. (b) Ambient decode 1,600 × 8 ms = 12.8 s: A = 2.64 + 12.8 + 4.75 = 20.2 s; C = 0.21 + 6.9 = 7.1 s → 2.8×. (c) A = 2.64 + 12.8 + 1.14 = 16.6 s → 2.3×. Prefix reuse (2.43 s) survives every scenario but is the smallest lever; the expedite lever shrinks as the ambient server improves; the round-trip lever shrinks as the client gets closer. The headline multiplier is scenario-dependent — which is exactly why papers report ranges and why you should distrust any single number, including ours.

Reading guide

Required — Parrot. Second read; read it differently. For Lecture 4 it described what an LLM application is; today it is a serving paper. Read the application characterization and the prompt-repeat measurements (the 3 / 94 / 72 / 99% figures) as workload evidence — this is the paper doing Lecture 5's instrumentation on production-style apps. Read the Semantic Variable design for precisely what crosses the API and what the server stores; read the scheduling material for how the DAG becomes objectives (which requests batch, which expedite) and how prefix sharing falls out structurally. Read §6 twice — dynamic control flow and native functions are the Assignment 2 loop, and the limits are this block's syllabus. In the evaluation, ignore the headline multipliers and extract the shape: where do gains come from (DAG-rich apps) and where do they vanish (chat-like, low-repeat apps)? Hold this question: which of the three payoffs would survive if every client already ran RadixAttention underneath?

Optional — SGLang. Read only the RadixAttention material: the radix tree over KV, LRU eviction, cache-aware scheduling. Skip the frontend language and constrained decoding. Hold: what does runtime recovery get for free that Parrot must ask the programmer for — and what can it never get, at any cleverness?

Optional — InferCept. Read for the interception taxonomy (tool calls, human turns, agent stalls) and the three-way cost model — discard, preserve, swap — as mechanism, not for the improvement numbers. Hold: which Lecture 5 property is this paper the answer to, and why does it need no DAG at all?

Optional — Autellix. Abstract and introduction only. Note what it promises: program-level scheduling without requiring the application to declare a static DAG — the point on the design axis past Parrot. Hold it for this class's second half, which starts from §18.7's axis.

Looking ahead

Today declared the structure. The rest of the block spends it, one resource at a time, and each strand picks up a row of §18.1's table that structure alone does not close.

Lecture 19 — Monday, November 23: "Tool stalls, interception, and the sandbox." §18.3 admitted that tool-call edges still stall even with the DAG on the server, and §18.7 priced one stalled session three ways (keep / recompute / swap) in a box we did not schedule. Monday that box is the lecture: InferCept's cost model in full, the interception taxonomy that makes stall duration a distribution rather than a number, and the second stateful thing nobody budgets for — the sandbox the tool actually runs in. Required reading is InferCept, so the optional pass you take tonight pays twice.

Lecture 20 — the same meeting, Monday, November 23: "Session state and agent memory." The 13.4 GB session becomes a storage-hierarchy problem: HBM, host DRAM, NVMe, and the fetch-versus-recompute line that decides which tier is worth keeping a session in. It also settles a debt from Lecture 4 — what the client's context policy does to the server's cache, where the cheapest policy in submitted tokens turns out to be the most expensive in computed ones.

Lecture 21 — also Monday, November 23: "Multi-agent workloads and performance optimization." §18.4's fan-out with the barrier put back: sixteen maps that must all finish, where the join makes the straggler — not the mean — the latency. Assignment 5 is due Dec 2.

Lecture 22 — the second half of today's class: "Scheduling agentic programs." Autellix and Teola, and the dynamic side of the axis — scheduling the programs Parrot's §6 gives up on: the adaptive loop, non-clairvoyant program-aware scheduling, fairness between a three-step chat and a two-hundred-step agent. Bring the §18.7 axis diagram; the discussion starts where today's honesty paragraph ends.

The final project (12% of the grade) is under way by today, with its report due December 9 — §18.5 is direct ammunition: structural knowledge of what will recur beats LRU guessing, and the 1/(1−r) arithmetic tells you where the points are, so spend an hour on it this week rather than in the last one. Nov 30 and Dec 2: the two Assignment 5 sharing sessions — half the room on Nov 30, the rest on Dec 2, the second closing with the course wrap-up — and the last meetings of the semester. Both come before the deadline: A5 is submitted the Sunday after, Dec 2. Assignment 5 puts you in the client seat of exactly today's client/server symmetry: the levers you will pull there are the ones today's papers pull from the other side of the API.