The agent-serving block improves the serving of agents by telling the system more: the DAG in this meeting's first half, and the interception type, return probability, and fan-out width on Nov 23. This half closes the axis from the other end and asks what a scheduler can do when it is told none of that — because the honest state of the world is that most clients will never declare anything, and the requests arriving at your engine right now carry no structure at all. Autellix's answer is that you do not need to be told: a program reveals its own length by consuming service, and a scheduler that accumulates service per program rather than per request recovers most of shortest-job-first without any clairvoyance. Along the way we will find that the API's choice to account by request rather than by program silently hands a 67× service-rate advantage to whoever issues more requests, which is a fairness bug hiding inside what looks like a neutral policy. Teola supplies the other half: a graph inside each request, which is the same idea one level down. By the end you should be able to place every mechanism in this block on a single granularity ladder and say what each one had to be told.
This is the second half of the meeting that opens the agent-serving block. The serving lectures from Sep 28 through Nov 11 built a serving stack whose unit of work was the request, because that is what the API delivers. This meeting's first half shows what declaring more buys; Nov 23 spends the declaration on stalls, state, and fan-outs. Today asks the residual question — what if nothing is declared? — and it is the practically important one, because every client that has not been rewritten is in exactly that position. Turning the relationship around — using LLMs to do systems research rather than doing systems research on LLMs — is optional content on this calendar rather than a block of its own.
Standing assumptions, unchanged through the block. Reference 7B, GQA-8, H100, ambient TPOT 22.6 ms, per-resubmission queueing Q = 200 ms (illustrative, from Lecture 18). Today's traffic mix, used throughout and stated once: 100 chat programs of 3 requests and about 200 output tokens total, plus 10 agent programs of 200 requests and about 16,000 output tokens total. That is 110 programs, 2,300 requests, and 180,000 output tokens — a mix in which agents are 9% of the programs and 89% of the work.
75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221). Instructor-led; the table is the plan for this half of the meeting. The instructor holds the framing, the §22.4 synthesis, and the block wrap, and leads the two paper deep-dives.
| Time | Segment | Notes |
|---|---|---|
| 0–8 | Framing: the unit of work (§22.1–22.2) | Instructor. The 199 × Q re-entry penalty and the traffic mix on the board before the deep-dives start. |
| 8–28 | Paper 1 deep-dive: Autellix (§22.3) | Protected. Must get to what statistic is accumulated and at what granularity. Instructor forces the "why is this not just SJF" question. |
| 28–40 | Synthesis: what non-clairvoyance costs (§22.4) | Protected — instructor-held. The 8.6× mean-flow figure and the MLFQ argument must be on the board when the bell goes. |
| 40–50 | Fairness (§22.5) | The 67× derivation live. Expect the room to argue; let them, it is the best discussion in the block. |
| 50–65 | Paper 2 deep-dive: Teola (§22.6) | Instructor. Push toward resource heterogeneity — why a graph and not a pipeline. |
| 65–73 | The granularity ladder and block wrap (§22.7) | Instructor. The whole block, one table. |
| 73–75 | Logistics: Assignment 5 due Dec 2, project report due Dec 8 | Announcement only. |
Reading-only, not scheduled: §22.8's discussion seeds beyond the first three, and the SLO decomposition in §22.5.
If running long: compress §22.6 to the primitive graph and one optimization; cut seeds 4–5. Never cut §22.4 or the ladder in §22.7 — the ladder is the only place the whole block is assembled in one view, and students will want it for Assignment 5.
By the end of this class you should be able to:
Define it precisely, because the whole lecture turns on the definition. A program is the set of model invocations belonging to one user task, connected by dependence: request k+1's prompt contains request k's output, or is decided by it. Lecture 5 characterized this stream; Lecture 18 called its shape a DAG and had the client declare it. Today we assume nothing is declared and the engine sees only what arrives: a sequence of independent-looking requests, some of which happen to be continuations of others.
Three properties of a program that no single request exposes.
Its length is unknown, and unknowable in advance. A chat program is three requests. A coding agent is two hundred, or twelve, depending on whether the tests pass. Lecture 18's §6 discussion made this point about declaration — dynamic control flow cannot be declared because the edge does not exist until the value does — and it applies with more force to scheduling, because a scheduler must act now.
Only its completion is user-visible. Lecture 18 established this for the sink of a DAG; it is the same fact. An intermediate request's latency is worth nothing to anybody, and any metric computed over requests is measuring something no user experiences.
It re-enters the queue once per step. This is the property with teeth, and §22.2 prices it.
Take a scheduler with no ill intent: plain first-come-first-served over arriving requests, which is what most engines approximate. A program of n steps submits its first request, waits, receives the output, and submits the second — which arrives now, behind everything that has arrived in the meantime, and waits again.
At Lecture 18's illustrative Q = 200 ms, a 20-step program pays 3.8 s and a 200-step program pays 39.8 s — before any of its actual work is done. The penalty is linear in program length, and no individual request was treated unfairly at any point. That is the shape of the problem: a policy that is fair to every request is systematically unfair to long programs, because length is measured in re-entries.
It gets worse under load. FCFS with continuous batching does not simply queue; it shares. Each request in the running set gets a slice, so a program's service rate is proportional to how many requests it has in flight — which for a sequential agent is exactly one, no matter how many it will eventually issue. Meanwhile a fan-out program with sixteen concurrent workers has sixteen. The accounting unit rewards concurrency, not need.
110 programs issue 2,300 requests: 100 chats × 3 = 300, and 10 agents × 200 = 2,000.
Under per-request fair sharing each request gets 1/2,300 of the engine, so a chat program's share = 3/2,300 = 0.13% an agent program's share = 200/2,300 = 8.70% An agent program receives 67× the service rate of a chat program.
Under per-program fair sharing each of the 110 gets 0.91%. Chats speed up 7×; agents slow down 9.6×.
In wall clock: a chat needing 4.52 s of exclusive service completes in 58 minutes under per-request sharing and 8.3 minutes under per-program sharing.
Nobody designed that 67×. It is an artifact of the unit the API happens to expose, and it is the exact analogue of a network scheduler that gives bandwidth per packet rather than per flow — where the fix, fair queueing, is fifty years old and required only that the scheduler learn to group packets by flow. The agent version needs the same thing: group requests by program. Which requires knowing which requests belong together, and that is where Autellix starts.
Minutes: 8 (with §22.1). Board: (n − 1) × Q, then 3.8 s and 39.8 s. Then the 0.13% / 8.70% / 67% column. Ask the room: "Which scheduling policy in this course was unfair to agents?" All of them. Then: "which of them knew it?" None — because none had a way to name a program. Expect confusion: Students read the 67× as a claim that agents are being favoured deliberately. It is emergent from the accounting unit, which is why it is interesting.
(Instructor-led. What follows is the framing the discussion should reach, not a substitute for it.)
Two problems have to be solved and they are separable.
Grouping. Which requests belong to the same program? This is the easy half in practice — a session identifier, an API key plus a conversation id, or simply the observation that this request's prompt contains the previous response verbatim. Note the last one: prefix matching, which Lecture 13 built for caching, is also a grouping signal. The radix tree already knows these requests are related; the scheduler was simply never shown.
Ordering. Given groups, in what order should they run? Shortest-job-first minimizes mean flow time and is unimplementable, since program length is unknown. The classical answer, and Autellix's, is to stop predicting and start observing: accumulate service per program, and demote a program as its total consumed service grows. New programs enter at high priority and finish fast if they are short; long programs sink through the levels and keep progressing at lower priority. That is multi-level feedback queue scheduling, applied at a granularity the engine did not previously have.
The three things the discussion should extract, in order of importance.
(Instructor-held synthesis. Protected.)
Bound the prize before evaluating any mechanism. Take the standing mix on a single server, with service measured in token-units, and compare the two extreme orderings.
Service: 100 short programs at 200 units, 10 long at 16,000 units.
Shortest-first. Shorts complete at 200, 400, …, 20,000 → sum = 200 × 5,050 = 1,010,000. Longs then run: completions 36,000, 52,000, …, 180,000 → sum = 10 × 20,000 + 16,000 × 55 = 1,080,000. Total 2,090,000 over 110 programs → mean = 19,000 units.
Longest-first. Longs complete at 16,000, …, 160,000 → sum = 880,000. Shorts then run from 160,000 → sum = 100 × 160,000 + 200 × 5,050 = 17,010,000. Total 17,890,000 → mean = 162,636 units.
Ratio: 8.6× between the best and worst orderings of the same work.
That 8.6× is the entire prize available to any program-aware scheduler on this mix, and it is worth noting what produced it: not faster kernels, not better batching, not a byte saved anywhere. The same work, reordered. It is also worth noting the direction — the 8.6× is available because the mix is skewed, with a few enormous programs among many tiny ones, and Lecture 21's tail discussion says agent traffic is reliably skewed like this.
MLFQ does not achieve the full 8.6×, because it pays to discover which programs are long. The mis-scheduling is bounded by the first quantum per program, and with 110 programs and a quantum small relative to 16,000 units the loss is a few percent of the gap — which is why non-clairvoyant scheduling is a good deal, and the number to keep is that most of an 8.6× is recoverable without knowing anything.
Two caveats that the discussion should surface rather than the notes.
Mean flow time is not the only objective. SJF minimizes it while maximizing the variance of the experience: the ten long programs wait behind everything. If your service level agreement is about tail program latency rather than mean, the optimal ordering changes, and §22.5's fairness question is the right frame.
A single server is a fiction. Real engines batch, so "ordering" is really "admission to the running set," and a program at low priority is not stopped but slowed. That is strictly better than the model above and makes starvation less acute, at the price of making the analysis harder.
Minutes: 12. Protected. Board: The two sums, then 19,000 versus 162,636, then 8.6× boxed. Underneath: "same work, reordered." Ask the room: "What fraction of the 8.6× would you give up to avoid starving the long programs?" There is no right answer; the question makes the fairness section necessary. Expect confusion: Students want to know which the engine "actually does." Most do neither — they do per-request FCFS, which is closer to random ordering than to either extreme, and random lands nearer the bad end because the long programs are large enough to block whatever follows them.
Three defensible definitions, three different systems.
Per-request fairness. Every request gets an equal share. This is what the API's unit implies and what most engines implement by default. §22.2 showed what it produces: a 67× service-rate advantage for programs that issue more requests. Almost nobody would defend it if it were stated out loud, and almost everybody deploys it.
Per-program fairness. Every program gets an equal share, regardless of how many requests it issues. Intuitive, and it is what a user would expect if told how the system works. But it means a 200-step agent and a 3-request chat receive the same capacity, so the agent takes 67× longer than it otherwise would — and if the agent is a paying customer's production workload and the chat is somebody idly asking a question, that is not obviously right either.
Per-token fairness. Every program gets a share proportional to work done, which is weighted fair queueing with tokens as the currency. This is the one that generalizes cleanly, prices correctly, and requires the least explanation to a customer — and it says nothing about latency, which is what people actually complain about.
The useful metric for comparing them is slowdown: a program's completion time divided by the service it required. A perfectly served program has slowdown 1; a chat that waits 58 minutes for 4.52 seconds of service has a slowdown of 770. Reporting mean slowdown rather than mean latency is what makes short and long programs comparable at all, and it is the metric the discussion should ask each paper to report.
SLO decomposition, for completeness and as reading. If a program carries an end-to-end deadline D over n steps, the naive decomposition gives each step D/n, which is wrong whenever steps differ — and they always do, since a step's cost depends on a context that grows. The better formulation schedules by laxity: D minus the predicted remaining work, smallest first, which is earliest-deadline-first adapted to unknown remaining length. It needs a length predictor, which returns us to non-clairvoyance, and the honest conclusion is that program SLOs are only as good as the predictor behind them — which is why Autellix does not offer them.
(Instructor-led.)
Autellix zooms out from the request to the program. Teola zooms in, and the observation is that what an application calls one request is already a small dataflow graph: embed the query, retrieve candidates, rerank them, generate an answer. The stages have different resource profiles — retrieval is CPU and memory bound, embedding and generation are GPU bound — and an opaque request forces them into a sequence on one timeline.
What the graph exposes, which the discussion should draw out:
Heterogeneous resources can overlap. While one query's retrieval runs on CPU, another's generation can run on GPU. Serialized within an opaque request, these alternate and each leaves the other resource idle; expressed as a graph across a batch of requests, they pipeline.
Stages have different batching behaviour. Embedding batches beautifully, retrieval batches differently, generation batches under all of Part II's constraints. One batching policy for the whole request is necessarily wrong for two of the three.
The critical path is visible. In a RAG request where generation is 2 s and everything else is 90 ms, optimizing retrieval is worth at most 4% — and knowing that requires the graph. Teola's contribution is less any single optimization than making the request's internal structure a first-class object so that such questions can be asked at all.
Which is the same contribution as Parrot's, at a different granularity, and the discussion should notice the pattern: every paper in this block improves things by making a structure explicit that was previously implicit, and they differ only in which structure and whose job it is to declare it.
(Instructor-held. The synthesis the block has been building toward.)
Arrange every unit of work this course has scheduled, smallest to largest:
token → request → primitive graph → program → workflow → fleet
Part II's earlier lectures lived entirely at the first two. The agent-serving block is the story of climbing the rest of the ladder, and each rung required somebody to know something.
| Lecture | Question | Unit | What the system must be told | Who tells it |
|---|---|---|---|---|
| 18 — Parrot | What crosses the API? | Request DAG | Templates and static edges | The application, explicitly |
| 19 — InferCept | What happens while nothing generates? | Interception | The tool name, and ideally the stall distribution | Nobody — it is already in the request |
| 20 — Continuum | Where does the state live? | Session | Probability of return | The application, or a predictor |
| 21 — TokenCake | What if k run at once? | Fan-join | Which child is critical, and the tail | The orchestrator, structurally |
| 22 — Autellix | Who goes next? | Program | Nothing. Service consumed is observed | Nobody |
| 22 — Teola | What is inside one request? | Primitive graph | The graph | The application, explicitly |
Read the fourth column down and the block's argument appears. It opened at maximum declaration — Parrot, where the application ships its structure — and closes at zero, with Autellix recovering most of the benefit from observation alone. In between, Lecture 19 finds a bit of structure that crosses the API already and is thrown away, and Lecture 20 finds one that nobody sends at all.
The conclusion is not that one end wins. It is that the agent workload contains parts best served at every rung, which is Lecture 18's honesty paragraph proven out by the rest of the block: a fan-out's shared prefix is best handled structurally, a tool stall is best handled by a free signal already present, and a program's length is best handled by not trying to know it. A serving system for agents is not a system that picks one of these. It is a system that does not throw away the ones it is given.
The block in one sentence: every improvement in this block came from naming a unit of work larger than the request, and the units differ in who has to say so — the application, the harness, or nobody at all. Prefer the mechanisms that need to be told nothing; they are the ones that will work on the clients you do not control.
k+1 shares a long prefix with request k's prompt-plus-output — which is nearly a proof that they belong to the same program. Should the prefix cache be the grouping mechanism for the scheduler? What breaks: two users of the same public system prompt, a program that compacts its context (Lecture 20's §20.4, which destroys the very prefix that identifies it), a fan-out whose sixteen children all match the parent equally well.(n − 1) × Q in pure queueing: 39.8 s for a 200-step agent at Q = 200 ms.| Quantity | Value | Source |
|---|---|---|
Queueing penalty, n-step program | (n − 1) × Q — 3.8 s at 20 steps, 39.8 s at 200 | Q = 200 ms, Lecture 18 |
| Standing mix | 110 programs, 2,300 requests, 180,000 output tokens | Agents 9% of programs, 89% of work |
| Per-request service share | chat 0.13%, agent 8.70% → 67× | 3/2,300 versus 200/2,300 |
| Per-program share | 0.91% each — chats 7× faster, agents 9.6× slower | 1/110 |
| Chat completion, per-request versus per-program | 58 min versus 8.3 min | 4.52 s of service |
| Mean program completion, best versus worst ordering | 19,000 versus 162,636 units = 8.6× | Shortest-first versus longest-first |
| Slowdown of that chat under per-request sharing | 770 | 3,465 s ÷ 4.52 s |
| Granularity ladder | token, request, primitive graph, program, workflow, fleet | §22.7 |
n steps rejoins the queue n − 1 times, paying Q each time, so its queueing cost is linear in a length the scheduler never sees. Every individual request was treated fairly; the program was not, and the scheduler has no object corresponding to the program in which the unfairness could be noticed.Required — Autellix. Read for the mechanism, not the numbers. First: what exactly is grouped, and how does the system decide two requests belong to the same program? Second: what statistic is accumulated, over what window, and what happens to it during a stall — this is the hinge, and the answer connects directly to Lecture 19. Third: the demotion and aging rules, which are the fairness policy whether or not the paper calls them that. In the evaluation, find the workload where the gains are smallest and say what property of that workload explains it; that is the most useful thing you can bring to the discussion. Hold this question throughout: what would Autellix do with Lecture 21's sixteen-way fan-out, and is that the behaviour you want?
Required — Teola. Read the primitive-graph representation and what it makes expressible that a request does not. Focus on resource heterogeneity — which primitives contend for what — because that is the source of the wins, and on how the graph is obtained, since somebody has to write it. Hold: is this a compiler, a scheduler, or an API, and does the paper's answer match yours?
Optional — SAGA. The workflow as the atomic scheduling unit, one rung above Autellix on §22.7's ladder. Read the argument for atomicity: what breaks when a workflow is partially scheduled, and is that argument stronger or weaker than Autellix's?
Optional — Helium and agentic AI workload characteristics. The first reframes agentic serving as a data systems problem, which is a genuinely different set of instincts and worth an hour. The second is the measurement study this block has needed throughout: check Lecture 21's assumed length distributions and today's traffic mix against it, and note every place we were wrong.
Part II closes next Monday. It opened with the observation that a decode step reads 13.5 GB of weights to produce one token; this half closes the declaration axis with a scheduler reordering programs it was never told about. The through-line is the one Lecture 1 promised: every improvement came from finding a unit of work the previous layer had thrown away — a shared prefix, a stalled session, a fork, a program.
Monday, November 23 (student-led): "Efficient agent serving systems II" — the rest of this block, spending the declaration on stalls, state, and fan-outs, and the last meeting of Part II. Assignment 5 (optimize the full stack) is due Dec 2, 11:59pm, and the final project report is due Dec 8, 11:59pm — for a project on prefix reuse, the whole of Lectures 13, 20, and 21 is ammunition, and §20.4's inversion in particular tells you what a trace's context policy implies about the reuse you can possibly extract.
Using LLMs to do systems research — the inverted relationship, models put to work on systems problems — is optional content rather than a lectured block; Lecture 24's notes cover the second half of it. Nov 30 and Dec 2 are the two student sharing sessions on Assignment 5 — what you optimized and what it cost — and Dec 2 is the last meeting of the semester. In Assignment 5 you are the client and this block's papers are the server — and Lecture 21's advice about the verifier is the most valuable sentence in the block.