Nov 18 put the request DAG on the server and deleted every LLM-to-LLM round trip, then admitted in one sentence that tool-call edges still stall — the server cannot run your Python. Today that sentence is the lecture. Half of an agent session's wall clock is spent generating nothing, and the session does not become free while it waits: it holds its KV cache, its scheduler slot, and its admission reservation, and on the CPU side it holds a second stateful object nobody budgets for — the sandbox its tools run in. We will price the stall in three currencies, derive the two break-evens that decide whether to keep, swap, or discard a waiting session's cache, and find that both thresholds are properties of the model and the card rather than of the session. Then we will overcommit deliberately, discover that the binding constraint moves from HBM to PCIe, and land on a throughput expression with a startling property: once the engine is swap-bound, task throughput does not depend on how long the tools take at all. By the end you should be able to look at a tool name and say what should happen to that session's cache.

Where this sits

Lecture 5 measured the stall and named it one of five properties of the agent request stream. Lecture 18 declared the structure and showed that declaration removes the dependence edges — the LLM-to-LLM round trips — while leaving tool edges exactly where they were. Everything Part II built before that assumed a request either occupies the engine or has left it; an intercepted request does neither, and that middle state is unbudgeted in every scheduler from batching and scheduling I (Oct 7) onward. Today is the middle state. The session-state part of today's meeting (Lecture 20) takes the same session and asks where its cache should live when the answer is "not here"; this part only asks whether it should still exist.

Instructor notes — Timing plan

75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221). This is the merged agent-serving meeting: the plan below is the full-class budget this note was written to, and it now has to be compressed to its share of a room it splits with session state and multi-agent workloads.

Time	Segment	Notes
0–6	Framing: the half of the session nobody schedules	Board the 30/30 split before saying anything else. "Nov 18 deleted the LLM-to-LLM edges. Today: the ones that are left."
6–14	§19.1 Duty cycle	Derive 50% live. End on "the session is idle; the memory is not."
14–22	§19.2 Three currencies	The 2.147 GB / 29 sessions / 3.44% chain. Keep it short — it is setup for §19.3.
22–42	§19.3 Three fates, two break-evens	Centerpiece — never cut. The 67 ms and 21.3 s thresholds must both be derived on the board, and the policy table must survive to the bell.
42–50	§19.4 Interception taxonomy	The distribution table against the two thresholds. Land the tie-back: the tool name is declared information in Nov 18's sense.
50–60	§19.5 The sandbox	Little's law live: 333 calls/s, 500 running, 667 booting. The pairing problem gets two minutes, no more.
60–70	§19.6 Overcommit	Protect. The 0.745 tasks/s expression and its independence from T. If only one thing survives from the second half, this.
70–75	§19.7 in brief + close	The 1.43× overlap number and one sentence pointing at the session-state part that follows.

Reading-only, not scheduled: §19.8 in full, the cold-start throughput corollary in §19.6 (state the result, assign the derivation), and the tool-result caching arithmetic in §19.7.

If running long: compress §19.5 to the boot-versus-run concurrency pair and drop the pairing problem; compress §19.7 to the overlap number. Never cut §19.3 — without both break-evens the lecture is a list of options with no rule for choosing between them.

Learning objectives

By the end of this class you should be able to:

1. Compute an agent session's duty cycle from a model/tool time split, and say what fraction of a card's KV capacity a stalled fleet denies.
2. State the three fates of an intercepted session's KV cache and price each one in the resource it actually consumes — HBM-seconds, GPU-seconds, or PCIe-seconds.
3. Derive both break-evens — keep versus swap, and keep versus discard — and explain why each is a property of the model and the card rather than of the session.
4. Explain why the correct policy is a function of the interception type, and why that type is available to the scheduler for free.
5. Size a sandbox pool with Little's law, and say when boot concurrency exceeds run concurrency.
6. Derive the throughput of a swap-bound overcommitted engine, and explain why it is independent of how long tools take.

19.1 The duty cycle: what fraction of an agent session is not generating

Start where Lecture 5's instrumentation ended. The 20-step designer task takes about 60 seconds of wall clock, split roughly evenly: 30 seconds of model time — prefill plus decode, summed across all twenty steps — and 30 seconds of tool time, during which the model produces nothing because it is waiting to be told what happened. Twenty steps, so twenty stalls, averaging 1.5 seconds each.

Half. That is the number the rest of the lecture argues with, and it is not an unusual one — Lecture 5 derived it from a real instrumented run and used it for an Amdahl argument (halving model latency buys at most 25% of end-to-end, because half the clock is not model latency). Today we ask the opposite question. Amdahl asked what the stall costs the user. Ask instead what it costs the server, and the answer is not symmetric, because the two halves consume different things.

During the 30 seconds of model time the session consumes compute and bandwidth, and it is in everybody's interest that it does — that is the engine working. During the 30 seconds of tool time the session consumes memory and nothing else. It generates no tokens, occupies no compute, and contributes nothing to any throughput metric, while continuing to hold every byte of KV cache it has accumulated. It is, from the engine's point of view, indistinguishable from a session that is about to generate — because nothing in the request stream says otherwise.

Here is the sharp version, and it is worth writing on a board. A chat request's memory occupancy and its usefulness are the same interval. An agent session's are not: occupancy is the whole session, usefulness is half of it. Every mechanism in Part II that reasoned about "requests in the batch" silently assumed the first case.

The rest of the lecture is about that parking lot: what it costs, what the alternatives cost, and how to decide per session which one to use.

Instructor notes

Minutes: 8. Board: The 30/30 timeline, twenty steps, one stall marked. Then the two-line contrast: "chat — occupancy = usefulness" over "agent — occupancy = 2 × usefulness." Ask the room: "During the stall, what is the session consuming?" Push past "nothing" to "memory." Then: "Which Part II mechanism accounted for that?" None of them. Expect confusion: Students conflate duty cycle with GPU utilization. A 50% duty cycle does not mean the GPU is half idle — other sessions fill the compute. It means half of this session's memory reservation buys nothing, and memory is the resource the KV-cache arithmetic (Sep 2) showed was binding.

19.2 What a stalled session denies, in three currencies

"Holds 2.147 GB" understates it, because the byte count is only one of three things a stalled session occupies. Name all three, because the fixes in §19.3 attack them separately.

HBM bytes. The obvious one. A 16,384-token session at GQA-8 holds 16,384 × 128 KiB = 2.147 GB, which against the 62.5 GB KV budget is 3.44% of the card. Twenty-nine such sessions fill the card exactly. Note how much better this is than Lecture 18's 13B-class MHA example at 13.4 GB — GQA and a smaller model bought a factor of six, which is the difference between a card holding 29 sessions and a card holding 4. Architecture choices show up here first.

A scheduler slot. Continuous batching (batching and scheduling I, Oct 7) maintains a set of active sequences and iterates over it. A stalled sequence either stays in the set — where it is examined every step and contributes nothing — or leaves it, which is a decision with consequences and is precisely what §19.3 is about. Engines that lack a "parked" state have only these two options, and most of them implement the first by default.

An admission reservation. Lecture 10 §10.1 made the point that admission is a promise issued without knowing how long the request will live, and §10.2 priced the safe answer: reserve for the maximum the request might need, and watch utilization fall to 15.2% on a workload whose sequences average a sixth of their slot. A stalled agent session is worse than a stalled chat request here, because its context is still growing: the reservation must cover not what it holds but what it will hold when it resumes and appends a tool result of unknown size. Conservative admission plus stalls is how an engine ends up at 40% memory utilization while rejecting requests.

Three currencies, and the fixes trade between them: keeping spends HBM, discarding spends GPU compute, swapping spends PCIe bandwidth. There is no option that spends nothing, which is why this is a scheduling problem and not an engineering oversight.

Instructor notes

Minutes: 8. Board: The three currencies as three columns; leave them up — §19.3's table fills them in. Ask the room: "Which of the three is hardest to give back?" The admission reservation, because giving it back means admitting you cannot guarantee the session can resume. Expect confusion: "Just evict it" sounds free. It is the discard option and it costs 0.731 s of GPU per resume, which §19.3 derives. Do not let the word "evict" smuggle in "for free."

19.3 Three fates for a stalled session, and the two break-evens

The centerpiece. When a session intercepts, the engine chooses among exactly three fates for its KV cache, and InferCept's contribution is to model all three rather than hard-code one.

Keep. Leave the KV in HBM. Costs zero compute and zero transfer; costs f of the card's KV budget for the whole stall.

Discard. Free the KV. Costs nothing during the stall; costs a full re-prefill on resume.

Swap. Copy the KV to host DRAM, free the HBM, copy it back on resume. Costs two transfers of PCIe bandwidth and frees the HBM for the middle of the stall.

Price each on the standing session — 7B, GQA-8, 16,384 tokens.

Now the two decisions. Take them in the order the engine faces them.

Keep versus swap. Swapping frees f of the card for almost the whole stall, at the price of 0.067 s of a resource — PCIe — that the decode loop is not otherwise using. If PCIe is uncontended, swapping dominates keeping the instant the stall outlasts the round trip:

Sixty-seven milliseconds. Below that, the transfer costs more time than the freed capacity is worth; above it, keeping is strictly worse than swapping whenever host DRAM has room and PCIe has headroom. Both conditions matter and §19.6 breaks the second one on purpose.

Keep versus discard. Discarding buys back f · T card-seconds of KV capacity and pays 0.731 s of GPU. To compare them we need one currency, so make the assumption explicit: the engine is KV-limited, which is the central result of Sep 2's inference-cost arithmetic — the ridge point wants 295 concurrent sequences and the KV cache allows 29 — so one card-second of freed KV capacity converts to roughly one card-second of decode throughput. Under that assumption:

Twenty-one seconds. Now expand T* symbolically, because the algebra says something the number alone does not:

The session length S cancels. B_kv / k = 62.5 GB ÷ 128 KiB = 476,837 tokens — the card's KV budget expressed in tokens — and T* is that number times the per-token prefill cost. The break-even is a property of the model and the card, not of the session. A 4,000-token session and a 64,000-token session cross over at nearly the same stall duration.

Nearly, because t_prefill is itself a function of S once the quadratic term is honest: at 4,096 tokens it is 31.60 µs and T* = 15.1 s; at 16,384 it is 44.63 µs and T* = 21.3 s. So the break-even drifts upward with context length — longer sessions are more worth keeping, because they are more expensive to rebuild — but it drifts linearly and slowly, and the first-order statement stands: this is a machine constant, and you can compute it once at deployment and hard-code it.

Change the architecture and it moves sharply. At MHA's 512 KiB per token the budget is 119,209 tokens and T* falls to 5.3 s at the same context length: four times less patient, because each session costs four times more to store and no more to rebuild. GQA does not only buy batch size — it buys the right to wait.

Assemble the policy:

Stall duration	Fate	Binding cost
under 67 ms	Keep	The swap round trip exceeds the capacity it frees
67 ms to 21.3 s	Swap	PCIe is cheap, HBM is not, recompute is 10.9× the transfer
over 21.3 s	Swap, or discard if PCIe or host DRAM is contended	Freed capacity now exceeds the rebuild cost

Read the middle row twice, because it is where the median tool call lives. Our standing stall is 1.5 s — twenty-two times the keep-versus-swap threshold and fourteen times below the keep-versus-discard threshold. For the typical tool call, swap is correct and both of the obvious policies are wrong. Keeping wastes 3.44% of a card for 1.5 s twenty times per session; discarding pays 0.731 s of GPU to avoid 0.067 s of PCIe, an 10.9× overpayment in the scarcer resource.

The honesty paragraph. The keep-versus-discard conversion — one card-second of KV capacity equals one card-second of throughput — is a linearization, and it is only defensible while the engine is genuinely KV-limited. An engine running well below capacity has free HBM, f · T buys nothing, and keeping is correct at every duration. That is not a flaw in the model; it is the model telling you that all three policies are correct somewhere, which is exactly why the paper builds a cost model instead of picking a winner. The 64 GB/s PCIe figure is illustrative and the whole middle row of the table depends on it: exercise 3 varies it and finds where the row disappears.

Instructor notes

Minutes: 20. Never cut. Board: Three fates, three prices, then the two break-evens derived in order. Box 67 ms and 21.3 s. Then write T* = t_prefill · B_kv / k under the second and circle the cancelled S. Ask the room: Before deriving T*: "Does a longer session cross over sooner or later?" Most say sooner — bigger cache, more urgent to free. Let them commit, then cancel the S. The surprise is the point. Ask the room: "Our stall is 1.5 s. Which fate?" Then: "Which fate does your engine actually implement?" Almost certainly keep, by default, silently. Expect confusion: Two. (1) Students want to add the swap and recompute costs to compare against keep; they are alternatives, not stages. (2) "Discard is free during the stall" gets read as "discard is free." The cost is deferred to resume, and it lands on the critical path of a user who is now waiting.

19.4 Interception is a taxonomy, not an event

Both break-evens are single numbers; stall durations are not. They are a mixture of distributions, one per reason the session stopped, and the distributions differ by four orders of magnitude. That is InferCept's central workload claim, and it is what makes a global policy indefensible.

Set the illustrative durations against the two thresholds derived above:

Interception type	Typical stall	Fate under §19.3
In-process function, arithmetic, string work	0.1–5 ms	Keep — far below 67 ms
Local file read	1–20 ms	Keep
Vector-store or index lookup	10–50 ms	Keep, marginally
Code execution, warm sandbox	50–500 ms	Swap
Code execution, cold sandbox	0.2–2.5 s	Swap
Web fetch or third-party API	0.2–5 s	Swap
Sub-agent call	5–60 s	Swap; discard past 21.3 s if PCIe is tight
Human turn	seconds to never	Discard

The table crosses both thresholds. A policy of "always keep" is right for the first three rows and wrong for the last five; "always discard" is right for the last row and catastrophically wrong for the first three, where it would pay 0.731 s of GPU to avoid holding memory for a millisecond.

Now the part that connects today to Nov 18. The interception type is available to the scheduler for free. When a session intercepts, the request carries the tool name — read_file, web_search, run_python, ask_user — because the model just emitted it and the harness just dispatched it. No Semantic Variable is required, no DAG, no API change: this is structural information that the classic completions API already carries and that every engine currently discards. Lecture 18's axis asked how much of the program crosses the wire; the answer here is that one very useful bit already does, and nobody reads it.

What the type gives you is a prior, not a value, and the difference matters because the tail is where the bill is.

Which is why a type prior is the floor rather than the ceiling: within run_python there are 50 ms calls and 200 s calls. Predicting the individual stall — from the tool name, the arguments, the session's own history — is what SpecBox does for sandbox prewarming and what Continuum turns into a retention TTL in the session-state part of today's meeting. Today's contribution is the decision rule that any predictor feeds.

Instructor notes

Minutes: 8. Board: The taxonomy table with the 67 ms and 21.3 s thresholds drawn as two vertical lines through it. The visual is the argument. Ask the room: "What does the engine know at the moment of interception that it is not using?" Drive to the tool name. Then: "Is that a Parrot-style API change?" No — and that is the point. Expect confusion: Students want the mean stall to determine the policy. Do the tail worked block if they resist; one 60 s call outweighing nineteen 1.5 s calls usually settles it.

19.5 The other stateful thing: the sandbox

Everything so far treats the tool as a black box that takes time. It is not a black box; it is a process, and for anything that executes model-written code it is an isolated process, because the code is untrusted by construction. That isolation has a startup cost, a memory cost, and — the part that surprises people — a state affinity that mirrors the KV cache exactly.

The isolation spectrum, cheapest to strongest: a plain process with seccomp and Landlock filters (Sandlock's position — no image, no root, no KVM, microsecond setup); a container (image pull and namespace setup, hundreds of milliseconds to seconds); a user-space kernel like gVisor (stronger confinement, syscall overhead, compatibility gaps); a microVM (Firecracker reports boot times under 125 ms and over a hundred microVM launches per second per host); a full VM (seconds). Every step right buys confinement and pays startup, and startup is a stall extension — which §19.3 just showed is the quantity that decides KV policy. The security decision and the memory-management decision are the same decision, taken by different teams.

Size the pool with Little's law, on the standing fleet.

Three readings of that block. First, per-call creation with containers spends more concurrency on starting than on working, which is the classic serverless cold-start result arriving in a new setting. Second, pooling trades boots for footprint — 1,000 idle-half-the-time sandboxes instead of 500 busy ones — which is the same keep-versus-rebuild tradeoff as §19.3, on the CPU side, with DeltaBox's checkpoint/restore playing the role of swap. Third, the two resources are not independent: a session's sandbox and its KV cache are both session-affine, and pooling one while discarding the other means the session resumes fast on the CPU and slow on the GPU, or vice versa.

That affinity creates the problem nobody plans for. The KV cache lives on a GPU node; the sandbox lives on a CPU node; the session is one thing spanning both. The migration story in routing and load balancing (Oct 26) assumed the migratable unit was KV. It is not: it is KV plus a live process with an open file descriptor table, a filesystem, and — if the interpreter is persistent across turns, which is the design most agent harnesses choose — variables the model expects to still be bound. Migrating the session means migrating or re-pairing both, and re-pairing is only free if the sandbox is stateless, which is a design choice with its own costs.

Instructor notes

Minutes: 10. Board: The isolation spectrum as a left-to-right axis with startup time under each point. Then Little's law: 333, 500, 42, 667. Ask the room: "Which end of the isolation spectrum should an agent platform pick?" There is no right answer; force them to name the quantity they are trading and notice it is stall duration — the same quantity as §19.3. Expect confusion: Students treat the sandbox as somebody else's problem — infrastructure, not serving. The link is that boot time extends the stall, and stall duration decides KV policy. Say it in that order.

19.6 Overcommit: admitting more sessions than fit

Now spend the analysis. If half of each session's lifetime is stall, and stalled sessions can be swapped out for 67 ms of PCIe, then the card should hold more sessions than its KV budget allows — because they are never all resident at once. Admit A sessions against 29 slots and let swap absorb the difference.

Two constraints bound A.

HBM. Only generating sessions need to be resident. At duty cycle d = 50%, resident ≈ A · d, which must not exceed 29 → A ≤ 58.

PCIe. Every stall now costs a round trip. Per session: 20 stalls × 2 × 2.147 GB = 85.88 GB per 60 s of session, or 1.431 GB/s sustained. Across A sessions that is A × 1.431 GB/s ≤ 64 → A ≤ 44.7.

PCIe binds first, and by a wide margin. Take A = 45.

Now generalize, because the general form has a property worth an entire board.

The session duration does not appear. Not because it cancelled by luck — because a swap-bound engine is limited by bytes moved per task, and a task moves the same bytes whether its tools take 100 ms or 100 seconds. The same cancellation runs through the resident count: generating sessions = (throughput × session duration) × d = throughput × model time = 0.745 × 30 = 22.4, independent of tool time as well.

The corollary is worth stating plainly because it contradicts the intuition everyone brings:

Two boundaries keep this honest.

Discard cannot substitute for swap here. Overcommitting with discard instead of swap costs 44.7 × (20/60) resumes/s × 0.731 s = 10.9 GPU-seconds per wall-clock second — infeasible by a factor of eleven. Overcommit is a swap technique specifically; the recompute path does not scale to one resume per session-step.

Overcommit does not always help. It beats resident-only admission exactly when

Our 60 s session clears it. A session that has been optimized — Lecture 18's Config C, where declared structure and prefix reuse cut model time to 7.1 s and total duration to 37.1 s — does not: its resident-only throughput is 29 ÷ 37.1 = 0.781 tasks/s, better than the swap-bound 0.745. Fix the serving path enough and swapping stops paying, because sessions now churn faster than PCIe can move their caches. Exercise 5 works this through.

Instructor notes

Minutes: 10. Protect. Board: The two constraints, 58 and 44.7, with PCIe circled. Then the throughput expression, then strike T out of it in front of them. Ask the room: "Tools got twice as slow. What happens to tasks per second?" Let them answer "halves," then derive that it does not change. The reaction is the lesson. Expect confusion: "Independent of duration" gets over-read as "stalls are free." They cost latency, they cost admission slots, and they cost the whole PCIe budget — which §19.8 and the session-state part of today's meeting both need for something else.

19.7 Shortening the stall instead of hiding it

Everything so far manages the stall. This section removes it, and opens with the observation that makes the removal urgent: the stall fraction grows as you fix everything else.

Nov 18's optimizations made today's problem four times larger in relative terms. That is not an argument against them; it is Amdahl reporting that the residual is now the majority. Four ways to attack it, in increasing order of how much they ask of the application.

Parallel tool calls. If a step issues three independent tools, issuing them serially costs 3 × 1.5 = 4.5 s and issuing them together costs about 1.5 s. This is free — the harness already has all three requests — and it is the single highest-leverage change most agent loops have not made.

Overlap execution with generation. The model emits a tool call token by token; the harness normally waits for the closing brace. If instead the arguments are streamed and execution starts as soon as they are complete — with a rollback path if the model changes its mind — the tool runs during the tail of the decode.

The honest caveat is that this is speculation, and it needs the same discipline as Lecture 4's retry rules: overlap freely for idempotent reads, never for a call with side effects, because a rolled-back post_invoice is not rolled back.

Cache tool results. Agent loops re-read the same files and re-issue the same queries constantly. If 30% of calls hit a result cache, 9 s of the 30 s stall disappears — and unlike the prefix cache, this one is the client's to build, because only the client knows which results are still valid.

Cap the output. Lecture 18's §18.8 called this the lever with no server-side twin, and it applies to time as well as tokens: a tool with no timeout is a stall with no bound, and the human-turn row of §19.4's table is what an unbounded stall looks like to the engine.

Instructor notes

Minutes: 5. Board: 56% / 50% / 19% in a column, then "the residual is now the majority." Ask the room: "Which of the four needs the serving system's cooperation?" Only overlap. The other three are yours, today, in the loop you wrote for Assignment 2.

19.8 What this changes about admission control

(Reading section — not scheduled in class.)

Admission control in batching and scheduling I (Oct 7) answered one question: is there room for this request's KV cache? Everything today says the question is under-specified for agents, in three ways.

Room for how long. A chat request's memory need is bounded by its output cap. An agent session's is bounded by nothing the engine can see: it grows by a tool result of unknown size at every step. Admitting a session is therefore a bet on its eventual footprint, and the conservative bet — reserve for the maximum context — is what drives the 40%-utilization deployments the inference-cost arithmetic (Sep 2) warned about. The practical answer is to admit against the current footprint and treat growth as an eviction event, which is only safe because §19.3's discard path exists.

Room where. With swap in play, "is there room" splits into HBM, host DRAM, and PCIe budget. The fleet in §19.6 is not HBM-limited at all; it is limited by a bus, and no admission controller written before agents existed asks about the bus.

Room for whom. Overcommitting to 45 sessions means 45 sessions can each experience a resume that finds no free HBM. The engine now needs a policy for resume admission — a queue for sessions coming back from a stall — and the fair thing is not obvious: a session that has already invested 40 s of work has a stronger claim than a fresh one, which is precisely the program-level fairness argument Lecture 22 took up on Nov 18.

A practical formulation that survives all three: admit on the stall-adjusted footprint, d · K rather than K, and hold the difference as a swap reservation rather than an HBM reservation. On our numbers that is 0.5 × 2.147 = 1.07 GB per session of HBM plus 1.43 GB/s of PCIe — which is exactly the 45-session answer §19.6 derived, arrived at from the admission side instead of the throughput side.

Discussion seeds

1. The conversion the paper needs and does not state. §19.3's keep-versus-discard break-even turns card-seconds of freed KV capacity into card-seconds of throughput, and that only holds while the engine is KV-limited. Construct a deployment on which keeping is correct at every stall duration, and say what you would measure to know which deployment you are running.
2. A prior is not a prediction. The tool name puts you in a regime; within run_python the calls run from 50 ms to minutes. Design the predictor you would actually ship, then price its two errors against each other: predicting short when it is long costs held HBM, predicting long when it is short costs 0.067 s of PCIe — or, if you discarded, 0.731 s of GPU on a user's critical path. Which error should your policy prefer, and why is that asymmetry not in the paper?
3. The optimization that turns its own mechanism off. Overcommit beats resident-only admission only above 38.9 s of session duration, and Lecture 18's Config C sits at 37.1 s. Should an engine detect that it has crossed its own threshold and disable swapping at runtime — and what would it have to measure to do that safely, given that the threshold moves with PCIe contention?
4. One session, two schedulers. The KV cache and the sandbox are both session-affine and owned by different teams on different nodes. Sketch the interface that would let them make one decision — prewarm-and-keep, checkpoint-and-swap, or destroy both — and say which side may veto the other.
5. Speculating on side effects. §19.7's overlap starts the tool before the model has finished asking for it, and buys 1.43× end-to-end. Write the rule that decides which tools are eligible. Is "idempotent read" a property the harness can verify, or one the tool author must declare — and if it must be declared, is that not Lecture 18's API-extension bargain again?
6. Who gets the resume slot. Overcommitting to 45 sessions means a resume can find no free HBM. A session 40 s into its task and one arriving now both want the slot. Which gets it, what does that do to the tail of the completion-time distribution, and how would you detect starvation?

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Duty cycle, instrumented 20-step task	50% (30 s model, 30 s tool)	Lecture 5
Session KV, 7B GQA-8, 16,384 tokens	2.147 GB = 3.44% of the 62.5 GB budget	16,384 × 128 KiB
Sessions per H100	29	62.5 ÷ 2.147
Prefill per token at S = 16,384	44.63 µs (27.26 weight GEMM + 17.37 attention)	361.5 TFLOP ÷ 494.5 TFLOP/s
Re-prefill on resume (discard)	0.731 s of GPU	16,384 × 44.63 µs
Swap round trip	0.067 s of PCIe	2 × 2.147 ÷ 64 GB/s
Keep-versus-swap break-even	67 ms	2K / BW
Keep-versus-discard break-even	21.3 s (5.3 s at MHA)	t_prefill · B_kv / k, S cancels
Sandbox concurrency, 1,000 sessions	500 running, 42 booting (microVM) or 667 (container)	Little's law at 333 calls/s
Swap-bound throughput	0.745 tasks/s, independent of tool time	64 ÷ 85.88 GB per task
Overcommit gain, and its condition	1.54×, only if session lasts over 38.9 s	0.745 vs 29 ÷ 60

Self-check

Exercises

1. Break-evens for a different deployment. A 70B model, GQA-8, d_model = 8,192, 80 layers, served across 2 H100s with a 100 GB aggregate KV budget; KV works out to 320 KiB per token and prefill to 140 µs per token at 8,192 tokens of context. Compute both break-evens and say which rows of §19.4's taxonomy move across a threshold. Solution sketch: Keep-versus-swap: session KV = 8,192 × 320 KiB = 2.68 GB, so 2 × 2.68 ÷ 64 = 84 ms — slightly higher than the 7B's 67 ms, so index lookups now fall safely on the keep side. Keep-versus-discard: budget in tokens = 100 GB ÷ 320 KiB = 305,175; × 140 µs = 42.7 s, double the 7B's 21.3 s. Bigger models are more worth keeping, because rebuilding them is superlinearly more expensive while storing them is only linearly so. The sub-agent row moves entirely onto the swap side.
2. The tail, properly. Twenty stalls drawn from a mixture: 80% at 0.3 s, 15% at 3 s, 5% at 90 s. (a) Mean stall. (b) Card-seconds of KV capacity consumed by "always keep." (c) By the §19.3 policy. (d) What fraction of the saving comes from the 5%? Solution sketch: (a) 0.8 × 0.3 + 0.15 × 3 + 0.05 × 90 = 0.24 + 0.45 + 4.5 = 5.19 s, versus a median of 0.3 s — the mean is a tail artifact. (b) 20 × 5.19 × 0.0344 = 3.57 card-seconds. (c) 16 calls kept (0.3 s each, under 67 ms? no — 0.3 s exceeds it, so swapped): all 20 swap, costing 20 × 0.067 = 1.34 s of PCIe and roughly 20 × 0.067 × 0.0344 ≈ 0.046 card-seconds of residual HBM. The one 90 s call is past 21.3 s, so discard it instead: 0.731 s of GPU. (d) The 5% contributed 4.5 ÷ 5.19 = 87% of the mean stall and therefore essentially all of the "always keep" bill.
3. Where overcommit stops paying. Holding everything else fixed, find the PCIe bandwidth at which swap-based overcommit admits no more sessions than fit resident, and the session duration at which it stops beating resident-only admission at 64 GB/s. Solution sketch: A = BW ÷ 1.431, so A = 29 at BW = 41.5 GB/s; below that, overcommit is strictly worse than doing nothing. At 64 GB/s the duration threshold is 29 × 85.88 ÷ 64 = 38.9 s. Both express the same fact: swap pays only when sessions are long and idle relative to the cost of moving their caches, and both of the ways to improve an agent system — faster tools, faster serving — push toward the region where it does not.
4. Sandbox pool under a burst. The fleet doubles for ten minutes: 2,000 concurrent sessions, same shape. (a) Boot concurrency needed at 2.0 s container start. (b) At 125 ms. (c) If prewarming keeps a pool of 800 idle sandboxes, what arrival rate can it absorb with zero cold starts, and for how long? Solution sketch: λ = 2,000 × 20 ÷ 60 = 667 calls/s. (a) 667 × 2.0 = 1,334 concurrent boots. (b) 667 × 0.125 = 83. (c) The pool absorbs the excess arrival rate over the replenishment rate; with 800 sandboxes and no replenishment it covers 800 ÷ 667 = 1.2 s of full-rate demand, which is the honest answer: a static pool is a shock absorber measured in seconds, not a capacity plan. Prewarming has to predict, which is SpecBox's argument.
5. Auditing overcommit against Lecture 18. Recompute §19.6 for Config C — model time 7.1 s, tool time 30 s, everything else unchanged. (a) Duty cycle. (b) Resident-only throughput. (c) Swap-bound throughput. (d) Which wins, and what does that say about optimizing the two halves independently? Solution sketch: (a) 7.1 ÷ 37.1 = 19.2%. (b) 29 ÷ 37.1 = 0.781 tasks/s. (c) unchanged at 0.745 tasks/s — it depends only on bytes per task. (d) Resident-only wins. Making the model half 6× faster moved the system out of the regime where swapping helps, so a deployment that adopted Lecture 18's optimizations and kept its swap-based overcommit would be slower than one that turned overcommit off. Optimizations interact; the two halves of the duty cycle are not separable, and any configuration you inherited was tuned for a duty cycle you may no longer have.

Reading guide

Required — InferCept. Read it in three passes with three different questions. Pass one, the taxonomy: what does the paper count as an interception, and what evidence does it give that the duration distributions differ by type rather than by request? That is §19.4's foundation and the paper's real contribution. Pass two, the cost model: reconstruct its discard/preserve/swap comparison and check it against the two break-evens we derived — you should be able to say which of our assumptions it does not make, and our KV-limited conversion in §19.3 is the first place to look. Pass three, the scheduling: how does the decision interact with continuous batching, and what does the engine do with a session whose predicted stall turns out wrong? Treat the improvement multipliers as its workloads' numbers, not as portable constants. Hold this question throughout: which of Lecture 5's five properties is this the answer to, and why does it need no DAG?

Optional — parallelizing tool execution and LLM generation. §19.7's overlap, done properly. Read for how it decides a tool call is complete enough to launch and what it does when the model contradicts the launched call. The speedup is not the interesting part; the rollback discipline is, and it is the same argument as Lecture 4's idempotency rules.

Optional — SpecBox and DeltaBox. The two halves of §19.5. SpecBox predicts and prewarms; DeltaBox checkpoints and rolls back. Read them as the CPU-side mirror of §19.3 — prewarming is speculative keeping, checkpointing is swapping — and ask whether the two sides should be making one decision instead of two.

Optional — Firecracker and Sandlock. Opposite answers to the isolation question, seven years apart. Firecracker buys VM-grade confinement and pays a boot; Sandlock argues that for short-lived agent commands the boot is the dominant cost and unprivileged kernel primitives are enough. Decide which you would deploy and be able to say what your answer assumes about stall duration.

Looking ahead

Lecture 20 — the second part of today's meeting: "Session state and agent memory." This part decided whether a stalled session's cache should exist; the next one decides where it should live. The three fates become a storage hierarchy — HBM, host DRAM, NVMe, remote — with one division deciding which tiers are worth using at all, and Lecture 13's fetch-versus-recompute break-even returns at session scale. It also settles a debt from Lecture 4: what the client's context policy does to the server's cache, where the policy that submits the fewest tokens turns out to compute the most.

Lecture 21 — the third part of today's meeting: multi-agent workloads. Today's stalls become simultaneous: when eight sub-agents run at once, the fan-out's barrier means the join waits for the slowest, and §19.4's tail stops being one call in twenty and becomes the thing every step waits for.

Lecture 22 — already covered on Nov 18, in the same meeting as the request DAG: scheduling agentic programs. §19.8's third question — who gets the resume slot — is a fairness question between programs, and Autellix answers it without being told how long any program will run.

Agent serving ends here, and so does Part II. Turning the relationship around — pointing the models at our problems instead — is optional content on this calendar rather than a lectured block; Lecture 24's notes and the readings are up if you want it. Assignment 5 (optimize the full stack, out Nov 11) is due Dec 2, 11:59pm, presented in class on Nov 30 or Dec 2, and the final project report is due Dec 8. Nov 30 and Dec 2 are the two student sharing sessions that close the semester.

One sentence to carry out of the room: an agent session's most expensive property is not what it computes but what it holds while computing nothing — and the cheapest predictor of how long it will hold it is the tool name the model just emitted and your engine just threw away.