CS2680 Modern AI Systems: Agents and System Optimizations

Readings

How to Use This Page

Every course reading in one place, grouped by class meeting and then by topic within the meeting: 389 unique works in 423 entries across 21 meetings. Repeated entries are intentional when one work serves more than one class.

The schedule on the home page remains the authoritative date-by-date view: it is the one to check the week before class, and if the two pages ever disagree, it wins. This page is the same axis with more on it. Each meeting below is split into the topics it covers, and under each topic sits everything the course has to say about that topic — the papers the lecture is built on, the background they stand on, the alternatives, and the follow-on work. One block is the whole of prefix caching, or speculative decoding, or tool stalls. Use it to prepare for a meeting, to choose a paper to present, or to find the background an assignment or a project needs.

Reading by Class below is the short version of the same thing: every meeting, its topics, and the papers each topic starts from. A work that belongs to more than one meeting carries an also note linking the others, and is listed under each of them.

You are not expected to read all of this, and no one will. A realistic plan: take the first paper or two under each topic for the meeting ahead, then one meeting’s block in depth for whatever you end up going deep on. If a topic is new to you, start from Surveys and Long-Form References rather than from the most recent paper in the list. The last group on this page, Not Tied to a Class Meeting, holds the self-study tracks and the reference material that no single meeting owns.

Presenting. Presentation papers are normally drawn from the topics listed for that date, but a well-argued substitution from the same meeting’s block is usually fine — ask first. The paper discussion page has the presenter guide, the rubric, and a three-pass reading strategy; Reading and Evaluating Systems Papers below has the two short pieces worth reading before your first paper.

Reading by Class

Every meeting, the topics it covers, and the papers each topic starts from. Follow a date to jump to that meeting’s block below, where each topic is expanded with the background, alternatives, and follow-on work behind it.

Class	Topic	Readings
Sep 2 Course overview and logistics	Why the course exists	Transformer, scaling laws, sustainable AI
Sep 9 Modern ML basics: transformers	Where inference cost comes from	efficiently scaling inference, Chinchilla
Sep 9 Modern ML basics: transformers	Frameworks	PyTorch, TensorFlow
Sep 14 Agents from a user’s perspective	The loop: reasoning and self-correction	ReAct
Sep 14 Agents from a user’s perspective	Context: retrieval and memory	RAG, MemGPT
Sep 16 Agents from a designer’s perspective I	The agent as a program the system can see	Parrot, SGLang
Sep 16 Agents from a designer’s perspective I	Tool use: teaching a model to call an API	Toolformer
Sep 21 Student sharing — what you learned from using an agent (Assignment 1)	Nothing to read. Bring your Assignment 1 transcript.
Sep 23 Agents from a designer’s perspective II	Interface, specification, and verification	SWE-agent, Reflexion
Sep 23 Agents from a designer’s perspective II	Multi-agent composition	AutoGen
Sep 28 GPU programming for MLSys: architecture, memory, and kernels	Microbenchmarking real hardware	dissecting Volta, dissecting Hopper
	Accelerator design and the programming model	TPU, CUDA programming guide
	The tile abstraction	Triton
	Hands-on	CUDA matmul walkthrough, Triton docs
Sep 30 Efficient LLM computing: GPU kernels	The performance model	roofline, making DL go brrr
	FlashAttention and its line	FlashAttention, FlashAttention-2, FlashAttention-3
	Attention kernels inside a serving engine	FlashInfer
Oct 5 Efficient LLM serving: paging, batching, and scheduling I	Iteration-level batching	Orca, NanoFlow, LoongServe
	Paging the KV cache	vLLM / PagedAttention, vLLM docs
Oct 7 Efficient LLM serving: batching and scheduling II	Interleaving both phases on one machine	Sarathi-Serve
	Splitting prefill from decode	DistServe, Splitwise, P/D-Serve
	Moving the KV cache between them	Mooncake, MemServe
Oct 14 Efficient LLM serving: routing and load balancing	Routing and live migration	Llumnix, Preble
	Multiplexing many models on a cluster	AlpaServe, MuxServe, DynamoLLM, Clipper
	Mixture of experts: the routing idea	GShard, Switch Transformer
Oct 19 Efficient LLM serving: KV-cache optimization I	Quantizing the cache	KIVI
	Evicting tokens	SnapKV
	Moving and accounting for the cache	CacheGen, KV cache management survey
Oct 21 Efficient LLM serving: KV-cache optimization II	A KV cache tier across the cluster	Mooncake, prediction-based KV management
	Reading less of the cache per token	Quest, InfiniGen, HiSparse
	Long context across devices	ring attention
Oct 28 Student sharing — what you optimized and what it bought (Assignment 3)	Nothing to read. Bring your Assignment 3 numbers.
Nov 4 Efficient LLM serving: prefix cache	Indexing shared prefixes	SGLang / RadixAttention, ChunkAttention
	Scheduling so the sharing pays	Preble, Parrot, Mooncake, MemServe
	Cache pools and hybrid models	Marconi
Nov 9 Efficient LLM serving: pruning and quantization	Outliers, and why naïve INT8 fails	LLM.int8(), SmoothQuant
	Number formats	mixed precision
	Post-training weight quantization	GPTQ, AWQ
	Rotations instead of special cases	QuIP#, QuaRot, SpinQuant
	Serving a quantized model at speed	QServe
Nov 11 Efficient LLM serving: speculative decoding	Guess ahead, verify in parallel	speculative decoding
	Where the guesses come from	EAGLE, EAGLE-2, Medusa
	Verification trees, long context, and scale	Sequoia, MagicDec
Nov 18 Efficient agent serving systems I	Declared structure: the request as a program	Parrot, SGLang
	Scheduling programs rather than requests	Autellix, SAGA
	End-to-end pipeline optimization	Teola, Helium
	What the workload looks like	agentic AI workloads
Nov 23 Efficient agent serving systems II	Tool stalls: what to do with the GPU state	InferCept, parallelizing tool execution
	Sandboxes fast enough to disappear	SpecBox, DeltaBox, Firecracker, Sandlock
	Sessions that outlive a request	Continuum
	Sharing state across agents	TokenCake, TokenDance, ForkKV
	Many agents at once	ScaleSim, large language monkeys
Nov 30 Student sharing — what you optimized and what it cost (Assignment 5), 1 of 2	Nothing to read. Bring your Assignment 5 numbers.
Dec 2 Student sharing — what you optimized and what it cost (Assignment 5), 2 of 2; course wrap-up	Nothing to read. Last meeting of the semester.

Part I — Introduction to LLMs and Agents

Sep 2 Course overview and logistics

Two topics for the first meeting. The first frames why the course exists: the architecture everything later optimizes, the scaling argument that made models large, and the energy and carbon bill that arrives afterward. The second is the capability results people usually have in mind when they say “LLM” — context for the rest of Part I.

Lecture notes for this meeting →

Paper	Year	Why read it
Why the course exists
Transformer — Attention Is All You Need	2017	The architecture the whole course optimizes. Read it for shapes, not for BLEU scores: which operations are matmuls, which are attention, and what has to be kept around from one token to the next.
scaling laws — for Neural Language Models	2020	Loss as a smooth power law in parameters, data, and compute. The reason anyone believed making models bigger would keep working.
sustainable AI — Environmental Implications, Challenges and Opportunities	2021	Carbon and energy accounting for the full lifecycle, including the parts usually left out.
What people mean when they say “LLM”
GPT-3 — Language Models are Few-Shot Learners	2020	Where in-context learning came from — and therefore why prompts got long enough that caching them pays.
InstructGPT — Training language models to follow instructions with human feedback	2022	How a base model becomes an assistant. The step that makes chat and tool use possible at all.
emergent abilities — of Large Language Models	2022	The claim that some capabilities appear discontinuously with scale — and a good exercise in asking whether an effect is real or an artifact of the metric.

Sep 9 Modern ML basics: transformers

Where inference cost comes from. Read Efficiently Scaling Transformer Inference closely: it is the analytic model the whole of Part II argues with, and its partitioning discussion sets up the serving lectures. The scaling-law group next to it is the live argument about the other half of that accounting — what a parameter count is worth, whether the compute-optimal recipe survives being checked, and why the axis has been moving from parameters toward data, precision, post-training, and compute spent at generation time. The rest of the block fills in the model itself — what a token is, which tensors get multiplied, what has to be kept between tokens, which architecture choices decide how big that state is, and which open-weight families the assignments run on.

Lecture notes for this meeting →

Paper	Year	Why read it
Where inference cost comes from
efficiently scaling inference — Efficiently Scaling Transformer Inference	2022	The paper to read closely. Analytic model of inference cost, partitioning strategies, and where latency actually goes at scale.
Chinchilla — Training Compute-Optimal Large Language Models	2022	The correction: models were badly undertrained for their size. It moved the optimum toward smaller models on more data, which is exactly why 7B-class models are worth serving.
The scaling-law argument
thoughts about scaling law — Jie Tang (Z.ai), on X	2026	The argument as practitioners are having it now, and the hypothesis the papers below are worth checking against. A parameter count means nothing on its own: it has to be read alongside how much data there was, where the compute goes per forward pass, and who will run the model under what conditions. Total parameters matter up to roughly “enough to hold the world,” and the dial with slack left in it is post-training — GLM-5.3 is offered as the controlled test, same base and same activated and total parameters as 5.2, one extra month of long-horizon RL. Note where that lands the bill: if capability now comes from post-training and from compute spent per forward pass, it lands on the serving system.
Chinchilla replication — Chinchilla Scaling: A replication attempt	2024	Refits Chinchilla’s own reported data and finds its headline parametric estimate inconsistent with that data, with confidence intervals far too tight to be credible. The roughly-20-tokens-per-parameter ratio broadly survives; the stated precision does not, and the paper’s three estimation methods do not agree as well as claimed. Worth reading straight after Chinchilla, as an exercise in how much of a scaling claim survives being checked.
over-training — Language models scale reliably with over-training and on downstream tasks	2024	Why nobody actually trains compute-optimal. Deliberately pushes models far past the Chinchilla token budget — which is what you do when you plan to serve the model for a long time and want it small — and shows loss and downstream error stay predictable there. This is the paper that connects the training-side scaling argument to the inference bill this course is about.
data limits — Will we run out of data? Limits of LLM scaling based on human-generated data	2022	Estimates the stock of public human-written text and projects when scaling the data term stops being an option. The reason the argument moved to post-training and to compute per forward pass is partly that one of the three terms has a floor.
scaling laws for precision — Scaling Laws for Precision	2024	Adds bits per weight to the scaling law and finds the terms interact: the more tokens a model saw, the more post-training quantization costs it, so the compute-optimal training precision is not 16 bits. Read it now for the shape of the claim and again before the quantization meeting, which is the systems half of the same question.
test-time compute — Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters	2024	The axis moves to inference: on some problem distributions, spending more compute at generation time beats spending it on a bigger model. That trade is the premise of the whole course — it converts a training-budget question into a serving-system question, and every optimization in Part II is about making the inference side of it cheap.
emergence, a mirage? — Are Emergent Abilities of Large Language Models a Mirage?	2023	The counterpoint to emergent abilities: a discontinuous metric can manufacture a discontinuity out of a smooth underlying curve, and swapping in a continuous one makes several famous jumps disappear. The transferable lesson is about measurement, not about emergence — the same mistake is available to anyone plotting a latency threshold.
observational scaling laws — Observational Scaling Laws and the Predictability of Language Model Performance	2024	Fits scaling laws across existing public models instead of training a new ladder, by placing them in a low-dimensional capability space. Useful if you ever want to make a scaling claim on a budget, which is the situation you will be in for the final project.
Frameworks
PyTorch — An Imperative Style, High-Performance Deep Learning Library	2019	Eager execution as a deliberate design choice, and what it costs.
TensorFlow — A system for large-scale machine learning	2016	The dataflow-graph alternative. Read it against PyTorch: the tradeoff is debuggability versus whole-program optimization.
Architecture choices that set the size of the state
RoPE — RoFormer: Enhanced Transformer with Rotary Position Embedding	2021	Rotary embeddings — why KV entries are position-encoded before caching, which constrains how cached prefixes can be reused.
ALiBi — Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation	2021	An alternative positional scheme aimed at extrapolating past the training context length.
SwiGLU — GLU Variants Improve Transformer	2020	Why the MLP block has three weight matrices instead of two — it changes the FLOP and parameter accounting.
RMSNorm — Root Mean Square Layer Normalization	2019	A cheaper normalization; a small paper that shows up in every fusion discussion.
Mamba — Linear-Time Sequence Modeling with Selective State Spaces	2023	State-space models: constant state per sequence instead of a growing KV cache. Read it for what a serving stack would look like without a KV cache at all.
Open-weight model families
BERT — Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	The encoder-only counterpart, and a useful contrast: no autoregressive decode, so no KV cache and none of the serving problems in Part II.
GPT-4 — Technical Report	2023	Almost no systems detail, which is itself the point: read it next to Llama 3 and DeepSeek-V3 to see how much closed reports withhold.
LLaMA — Open and Efficient Foundation Language Models	2023	The open-weight line the assignments run on.
Llama 2 — Open Foundation and Fine-Tuned Chat Models	2023	Adds the chat/RLHF half and the license that made open-weight serving mainstream.
Mistral 7B	2023	A 7B-class model in exactly the size range Assignment 4 targets; sliding-window attention.
Qwen2.5 — Technical Report	2024	A strong open-weight family with a wide size ladder — convenient when a project needs to sweep model size.
OLMo — Accelerating the Science of Language Models	2024	Fully open: data, code, checkpoints, logs. The one to pick if a project needs to see inside training.
DeepSeek-R1 — Incentivizing Reasoning Capability in LLMs via Reinforcement Learning	2025	Reasoning models emit far more decode tokens per request, which shifts every tradeoff in Part II toward the decode side.

Sep 14 Agents from a user’s perspective

What an agent is, before any of it becomes a systems problem: a loop that calls the model many times, retrieval to get facts in, and memory that has to outlive the context window. Read these for the shape of the workload rather than the prompt tricks — every one of these patterns turns into repeated, nearly identical prefixes later. The evaluation entries matter for Assignment 1: they decide what “it worked” means. The last group is the deliberate exception, a set of practitioner guides on driving a coding agent, which is what Assignment 1 asks you to do; it is ordered by how much each repays the time rather than by date.

Lecture notes for this meeting →

Paper	Year	Why read it
The loop: reasoning and self-correction
ReAct — Synergizing Reasoning and Acting in Language Models	2022	The interleaved reason/act loop that nearly every agent framework is a variation on.
chain-of-thought — Prompting Elicits Reasoning in Large Language Models	2022	Buying accuracy with decode tokens — the first place quality and cost trade off explicitly.
self-consistency — Improves Chain of Thought Reasoning in Language Models	2022	Sample many chains, vote. Linear cost increase for a bounded accuracy gain; a clean target for batching.
Tree of Thoughts — Deliberate Problem Solving with Large Language Models	2023	Search over reasoning states. Branching turns one request into a tree with a shared prefix — exactly the structure RadixAttention exploits.
Reflexion — Language Agents with Verbal Reinforcement Learning also Sep 23; in the agents track	2023	Self-critique across attempts, which makes trajectories long and stateful.
Self-Refine — Iterative Refinement with Self-Feedback	2023	The same idea at single-response granularity: generate, critique, revise.
Context: retrieval and memory
RAG — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks	2020	Retrieval-augmented generation — the origin of the long, partly repeated prompts that make prefix caching worth building.
MemGPT — Towards LLMs as Operating Systems also Nov 23	2023	Treats the context window as a memory hierarchy with explicit paging. The most systems-flavored agent paper on the list.
lost in the middle — How Language Models Use Long Contexts	2023	Long context is not free accuracy. Relevant when a project’s answer to a problem is “put more in the prompt”.
context rot — Diagnosing and Mitigating Context Rot in Long-horizon Search	2026	Measures accuracy degradation as accumulated search context grows, making the case that unbounded context growth is a correctness problem before it is ever a memory-capacity problem.
is GraphRAG needed — Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization	2026	Implements and compares regular, graph, modular, and agentic RAG across nine standardized scenarios on semi-structured knowledge bases. Reports a retrieval-generation gap where expanded retrieval does not proportionally improve generation quality, plus a context-engineering method that cuts token usage 19-53%.
Benchmarks, and how agents fail
WebArena — A Realistic Web Environment for Building Autonomous Agents	2023	Agents in a reproducible web environment with real applications behind it.
AgentBench — Evaluating LLMs as Agents	2023	A multi-environment agent benchmark; useful mostly as a survey of what people think agents should be tested on.
GAIA — a benchmark for General AI Assistants	2023	Questions that are easy for people and hard for assistants, which is a sharper test than it sounds.
METR time horizons — Measuring AI Ability to Complete Long Software Tasks	2025	Defines a 50%-task-completion time horizon — the human task duration at which a model succeeds half the time — and finds it has roughly doubled every seven months since 2019. A capability axis measured in task length rather than pass rate, which is the right shape for reasoning about how long agent sessions run.
BrowseComp — A Simple Yet Challenging Benchmark for Browsing Agents	2025	1,266 short-answer questions that are trivial to verify but require persistent, hard-to-predict multi-step browsing, so run length varies widely per query. The authors note it deliberately sidesteps a true user query distribution, so read it as a persistence probe rather than a workload model.
overthinking — The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks	2025	Introduces an overthinking score for agents that favor internal reasoning chains over environment interaction, and shows higher scores correlate with lower SWE-bench Verified performance across 4,018 trajectories. Selecting the lower-overthinking solution improved performance by almost 30% while cutting compute cost 43%, so more thinking is not monotonically better.
agent failure taxonomy — Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents	2026	Synthesizes 27 benchmark, taxonomy, and audit papers into six failure clusters — tool invocation, planning, long-horizon context accumulation, multi-agent coordination, safety, and measurement validity — instead of reporting aggregate pass rates. The claims that matter here: failures compound nonlinearly with task length, and additional scaffolding does not consistently improve reliability.
Driving a coding agent: practitioner guides
Claude Code best practices — Best practices for Claude Code (Anthropic)	—	The workflow it recommends has four phases: explore, plan, implement, commit. Two of its other rules matter more for this course. First, give the agent a check it can run, such as a test, a build, or a screenshot to compare against a design, because without one “looks done” is the only stopping signal the agent has. Second, treat the context window as the binding resource and clear it between unrelated tasks. Read the whole page as a workload description: plan mode, subagents for investigation, and fanning out over claude -p all change how many model calls a session makes and how much prefix those calls share, which is the workload Assignment 1 generates.
agentic engineering patterns — Agentic Engineering Patterns (Simon Willison)	2026	A guide in progress, one chapter per pattern, in the format the 1994 Design Patterns book established. The chapters cover what changes once writing code is cheap, red/green TDD as a loop the agent can close without you, verification, subagents, and version control as the undo mechanism that makes an unattended run safe. The author adds one or two chapters a week, so read it as a snapshot of current practice rather than a settled account.
how OpenAI uses Codex — How OpenAI uses Codex	—	Compiled from interviews with OpenAI engineers and internal usage data, so it reports what experienced users do rather than what a vendor recommends. Five habits: plan in Ask mode first and feed that plan to Code mode, scope each task to roughly an hour of human work or a few hundred lines, write the prompt the way you would write a GitHub issue (file paths, component names, diffs), keep persistent repository context in AGENTS.md, and use Best-of-N to generate several attempts and keep the best. Best-of-N is the one with a systems consequence, since it turns one task into several requests that share almost all of their prompt.
writing a spec for an agent — How to write a good spec for AI agents (Addy Osmani)	2026	The most careful treatment of task specification on this list: state goals before mechanics, organize the spec the way a product requirements document is organized, and replace a list of prohibitions with three tiers (always, ask first, never). It names the failure it is designed around, the curse of instructions, which is the drop in compliance as the number of requirements packed into one prompt grows. Its answer is decomposition into modular prompts plus a self-check the agent runs against the spec, and both are the same move the long-context entries above argue for from the measurement side.
an LLM coding workflow — My LLM coding workflow going into 2026 (Addy Osmani)	2026	One practitioner’s end-to-end routine in eight stages, from spec and plan through small tasks, model choice, tests, review, and frequent commits as checkpoints. Worth reading next to the spec guide above because it shows the same advice as a habit somebody actually follows rather than a list of principles, including where they hand work to a review bot instead of reading it.
coding with agents — Best practices for coding with agents (Cursor)	2026	Written from inside a different harness, which is what makes it worth reading alongside the Claude Code guide: the advice that appears in both is about agents, and the advice that appears in only one is about a product. It defines the agent harness (the loop, tools, and context assembly wrapped around the model) and covers plan mode, static rules against loaded-on-demand skills, parallel agents run to compare approaches, and a debug mode that instruments the code and collects runtime data instead of guessing.
context engineering — Effective context engineering for AI agents (Anthropic)	2025	Context engineering. Curating which tokens occupy the window at each turn, as opposed to wording one prompt well. The argument is that context is a finite resource whose value degrades as it fills, so the four techniques it names are all mechanisms rather than phrasings: compaction, structured note-taking into external memory, subagents that each get a clean window, and just-in-time retrieval that holds identifiers and loads the data only when a step needs it. Each one changes what the serving system underneath actually sees, which is why this sits next to context rot and lost in the middle above.
harness engineering — Harness engineering: leveraging Codex in an agent-first world (OpenAI)	2026	Reports a five-month internal project that shipped a beta product of roughly one million lines with no hand-written source, and argues that the engineer’s job shifts from writing code to building the environment the agents run in: tools, abstractions, tests, documentation, CI, and observability, so that an agent succeeds by default. The harness it names is the execution system between the model and the task, which gathers context, invokes tools, enforces sandbox and approval boundaries, and carries state across turns, and it is the same component Assignment 1 asks you to build a small version of. Note that the evidence is one team and one product, so treat the line count as an existence proof rather than a rate anyone should plan against.

Sep 16 Agents from a designer’s perspective I

The first paper that treats an agent as a program the serving system can see. Read Parrot for its central claim — if the system knows the structure of a request, it can schedule it better — and note how little of that structure a plain chat API exposes. The topics that follow are the tool-use line: how a model learns to call an API, what the interface itself costs once there are thousands of them, and what to do when the context stops fitting.

Lecture notes for this meeting →

Paper	Year	Why read it
The agent as a program the system can see
Parrot — Efficient Serving of LLM-based Applications with Semantic Variable also Nov 4, Nov 18	2024	Semantic Variables expose the dependency structure of a multi-call application to the scheduler, so it can optimize end-to-end latency instead of per-request latency.
SGLang — Efficient Execution of Structured Language Model Programs also Nov 4, Nov 18	2023	A front-end language plus a runtime, co-designed. Read it here for the programming model; the RadixAttention half is in prefix caching.
XFlow — An Executable Protocol Programming System for Reliable Multi-Agent Workflows	2026	Draws the prompt-harness boundary explicitly: workflow commitments move out of prompts into a compiled protocol DSL, where typed, lifecycle-governed state cells validate and mediate actor outputs before they become shared state. The design point opposite to free-form ReAct loops, trading agent autonomy for coordination that can actually be enforced.
Tool use: teaching a model to call an API
Toolformer — Language Models Can Teach Themselves to Use Tools	2023	Teaching a model when to call a tool, rather than prompting it to.
Gorilla — Large Language Model Connected with Massive APIs	2023	Tool calling when the tool catalog is far too large to fit in the prompt.
ToolLLM — Facilitating Large Language Models to Master 16000+ Real-world APIs	2023	Scaling tool use to thousands of APIs, with the retrieval step that implies.
HuggingGPT — Solving AI Tasks with ChatGPT and its Friends in Hugging Face	2023	An LLM as a planner that dispatches to other models — heterogeneous inference behind one request.
Voyager — An Open-Ended Embodied Agent with Large Language Models	2023	A long-running agent that accumulates a skill library — open-ended sessions rather than one-shot requests.
CoALA — Cognitive Architectures for Language Agents	2023	A framework for comparing agent designs instead of collecting them. Useful when writing Assignment 2.
MCP, and what an interface costs
MCP specification — Model Context Protocol Specification (2025-11-25)	2025	The wire-level definition of how tools, resources, and prompts are exposed to a model over JSON-RPC, including stateful sessions, capability negotiation, and server-initiated sampling. Read it as a systems interface spec and ask what per-tool schemas and session state cost you in every prompt.
MCP vs CLI — The Scaffolding Matters More Than the Interface: A Controlled Comparison of MCP and CLI Tool Use	2026	Runs one fixed software task across seven agent scaffoldings and five models and finds the scaffolding, not the MCP-versus-CLI interface, dominates cost; paired MCP-to-CLI ratios span 0.43x to 29x. It also finds agents frequently ignore the interface they were assigned, so any comparison that does not verify actual behavior measures an unknown mixture.
MCP systems cost — Network and Systems Performance Characterization of MCP-Enabled LLM Agents	2025	Measures the network, latency, and token overhead MCP imposes per tool call — the tradeoff between protocol generality and the per-request context tax it adds to every prefill.
MCP in practice — An Empirical Study of Model Context Protocol Applications	2026	A study of 1,723 MCP-consuming applications mined from GitHub: most configure servers through files and use an official SDK, but only 37.2% gate tool execution behind a blocking human approval, leaving the model free to invoke any enabled tool. Evidence on what the protocol leaves unspecified on the client side.
Managing the context window
context pruning — Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents	2026	Aggressive pruning of tool output and history can raise task success while cutting tokens, so context management is a joint accuracy and cost knob rather than pure compression.
HANDBOOK.md — A Benchmark for Long-Context Agentic Instruction Following	2026	A benchmark of 65 agentic tasks where a 20-124 page standing policy document is supposed to govern every action over a long tool-use horizon; the strongest evaluated model passes only 36.2% of trials under strict grading, dropping rules over long horizons and letting in-environment requests override policy. Concrete evidence on whether a large persistent instruction prefix is actually honored.

Sep 21 Student sharing — what you learned from using an agent (Assignment 1)

Nothing to read for this meeting. Bring your Assignment 1 agent transcript — one task it handled well, one it failed, and your read on why.

Sep 23 Agents from a designer’s perspective II

The second half of designing an agent, and the half that decides whether the first half worked. Everything here is about the three artifacts you write rather than the model you call: the specification the agent is held to, the verifier placed inside the loop so a bad step is caught while its error is still local, and the frozen task set that is the only instrument able to say whether a change was an improvement. Start from SWE-agent: it makes the argument with measurements — same model, better interface, different system. The multi-agent entries are here for the composition vocabulary and the return contract between agents. What the design you end up with does to the machine underneath is folded below, and is Part II’s subject rather than this meeting’s.

Lecture notes for this meeting →

Paper	Year	Why read it
Interface, specification, and verification
SWE-agent — Agent-Computer Interfaces Enable Automated Software Engineering also in the agents and LLMs-for-systems tracks	2024	The design paper of Part I: hold the model fixed, change the interface the agent acts through, and the success rate moves. Read it for the specific interface decisions — and for how they were evaluated, which is the harness problem of §5.3 solved at paper scale.
Reflexion — Language Agents with Verbal Reinforcement Learning also Sep 14; in the agents track	2023	The verifier loop closed automatically: the critique is written in language and re-enters the context. Price it with the lecture’s arithmetic — the critique is tokens per attempt riding in every later prompt.
Multi-agent composition
AutoGen — Enabling Next-Gen LLM Applications via Multi-Agent Conversation also in the agents track	2023	Multi-agent conversation as a programming model. The vocabulary for deciding when a second agent earns its bill — and, read against the fan-out arithmetic, for what the arrangement costs.
MetaGPT — Meta Programming for A Multi-Agent Collaborative Framework	2023	Role-structured multi-agent pipelines with explicit hand-offs — the return contract between agents made into the design.
generative agents — Interactive Simulacra of Human Behavior	2023	Many agents, each with memory and reflection. Read it for the memory design — what gets persisted, what triggers a fetch, and what happens when the store is stale.

Paper	Year	Why read it
Runtime substrates for agent programs
Ray — A Distributed Framework for Emerging AI Applications	2017	The distributed task model underneath a lot of agent and RL infrastructure.
InferCept — Efficient Intercept Support for Augmented Large Language Model Inference also Nov 18, Nov 23	2024	What to do with GPU state while a request is blocked on a tool call: keep it, discard it, or swap it out.
Teola — Towards End-to-End Optimization of LLM-based Applications also Nov 18	2024	End-to-end optimization across the whole application dataflow rather than the model call alone.
What the workload looks like from underneath
agentic AI workloads — Agentic AI Workload Characteristics also Nov 18	2026	A characterization of agentic workloads — the trace-level view of what these systems actually receive.
agentic workload characterization — From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems	2026	Instruments ten agentic applications plus production traces and names six ways they break conventional serving assumptions: non-LLM components dominate latency in half the applications, sessions hold state idle for minutes to hours between steps, and tool schemas plus auxiliary LLM calls levy a control-plane tax. Its design explorations — state offloading, tool-result caching — show the characterization is actionable.
TraceLab — Characterizing Coding Agent Workloads for LLM Serving	2026	About 4,300 real Claude Code and Codex sessions released as a trace, characterized on what serving cares about: long autonomous loops, long contexts with short outputs, heavy-tailed tool calls, and high but imperfect prefix cache hit rates. The trace artifact to drive your own simulation and cache-policy experiments.
CacheWise — Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents	2026	Measures coding-agent KV reuse first, then designs around it: prefix-aware scheduling plus reuse-aware eviction driven by lightweight predictions from tool-call metadata, implemented in vLLM. The concrete case that conventional serving policies mishandle the sustained KV pressure of long-running agent sessions.
architectural implications — Architectural Implications of Agentic AI Workflows	2026	An architectural characterization of agentic workflows from a production study at Microsoft Azure plus open-source frameworks: requests fragment into LLM calls, tools, and orchestration that repeatedly cross the CPU-GPU boundary, putting the CPU on the critical path and stranding both CPU and GPU capacity under bursty load. Useful for arguing which inefficiencies are schedulable and which are structural to uniform servers.
Where the waste goes
tool-use inefficiency — Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning	2026	Introduces Prefill Token Equivalents, a hardware-aware efficiency metric that charges what token counts miss — tool-call pauses that evict KV cache and force recompute, plus bloated tool responses — and uses it to name four recurring inefficiency patterns across five tool-use benchmarks. Also finds higher-cost trajectories tend to be less correct, so more tool use is not better.
energy per workflow — Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows	2026	Measures latency and energy together across four multi-request workflow patterns (sequential, interactive, agentic, composite) on an A100 running vLLM and Parrot, finding batch size the dominant knob but its payoff workload-dependent, and power capping a modest but predictable saving. Read it for where the energy-efficient configuration stops coinciding with the fast one.

Part II — Systems for LLMs

Sep 28 GPU programming for MLSys: architecture, memory, and kernels

The machine underneath every number in Part II, and the one meeting that opens it — architecture and the memory hierarchy first, then writing kernels against them (continued below). The lecture carries the material; the microbenchmarking papers are how you find out what the vendor documentation leaves out: real latencies, real cache sizes, real tensor-core behavior. Read them for the method, not for the specific numbers, which age with each generation.

Paper	Year	Why read it
Microbenchmarking real hardware
dissecting Volta — Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking	2018	Microbenchmarking a GPU until its actual memory hierarchy falls out. The method matters more than the specific numbers.
dissecting Hopper — Benchmarking and Dissecting the Nvidia Hopper GPU Architecture	2024	The same treatment for Hopper, including the tensor cores and features the LLM stack depends on.
Accelerator design and the programming model
TPU — In-Datacenter Performance Analysis of a Tensor Processing Unit	2017	A machine designed for one workload. Read it for the argument about what a datacenter accelerator is for.
CUDA programming guide — CUDA Programming Guide	—	Reference, not reading. Know where the memory-hierarchy and occupancy sections are.

GPU programming, continued: writing kernels — Triton, tiling, and fusion

Folded into the Sep 28 meeting above. This was a second meeting on an earlier version of the schedule; it is now the back half of one. Writing a kernel rather than reading about one: the tile abstraction is the whole idea — describe blocks and let the compiler place threads. Start with the hands-on entries — the matmul walkthrough and the Triton tutorials, which are the point of the meeting and worth an evening — then the paper for where the abstraction came from. The last two entries are supplementary: how far tiled kernel DSLs have been pushed since, for anyone taking the kernel-level path in Assignment 4.

Paper	Year	Why read it
The tile abstraction
Triton — An Intermediate Language and Compiler for Tiled Neural Network Computations	2019	The tile-level abstraction: write blocks, let the compiler handle threads.
TileLang — A Composable Tiled Programming Model for AI Systems	2025	Composable tiled programming, separating dataflow from schedule.
Hands-on
CUDA matmul walkthrough — How to Optimize a CUDA Matmul Kernel step by step	2022	Ten iterations from naive to near-cuBLAS, each with the reason it helped. The best single hands-on introduction.
Triton docs — Triton documentation	—	Tutorials first; the fused-softmax and matmul examples cover most of what Assignment 4 needs.
ThunderKittens — Simple, Fast, and Adorable AI Kernels	2024	Small tile primitives that make hand-written kernels tolerable to write.

Sep 30 Efficient LLM computing: GPU kernels

Where the arithmetic actually happens. The recurring theme: the win comes from moving less data, not from doing less math, so tiling, fusion, and where a tensor lives matter more than the FLOP count. Read the roofline paper first — it gives you the units for the rest of the course — then FlashAttention as the canonical application. The hardware and kernel-writing background is the GPU programming meeting before this one.

Paper	Year	Why read it
The performance model
roofline — An Insightful Visual Performance Model for Multicore Architectures	2009	The model that tells you whether you are compute- or bandwidth-bound before you optimize anything. The right first step in Assignment 5.
making DL go brrr — Making Deep Learning Go Brrrr From First Principles	2022	Compute, bandwidth, or overhead — decide which one you are fighting before you touch anything.
dissecting Blackwell — Dissecting the NVIDIA Blackwell Architecture with Microbenchmarks	2025	The Blackwell entry in the Volta/Turing/Hopper microbenchmarking lineage: measured latencies, throughputs, and cache behavior for the 5th-gen tensor cores (including FP4/FP6), execution pipelines, and memory hierarchy you need before trusting any Blackwell performance claim — note it measures a consumer GeForce RTX 5080 against an H100 PCIe, not a B200.
FlashAttention and its line
FlashAttention — Fast and Memory-Efficient Exact Attention with IO-Awareness	2022	Tile attention so the N×N score matrix is never written to HBM. The clearest demonstration in the course that IO, not FLOPs, is the budget.
online softmax — Online normalizer calculation for softmax	2018	Two pages, and the numerical trick that makes FlashAttention possible. Read it first.
FlashAttention-2 — Faster Attention with Better Parallelism and Work Partitioning	2023	Better work partitioning and fewer non-matmul FLOPs.
FlashAttention-3 — Fast and Accurate Attention with Asynchrony and Low-precision	2024	Asynchrony and FP8 on Hopper — the same algorithm rewritten for a new memory and execution model.
FlashAttention-4 — Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling	2026	The Blackwell successor to FlashAttention-3, where tensor-core throughput grew but SRAM bandwidth and the exponential unit did not; a concrete lesson in redesigning a kernel when hardware scales asymmetrically.
FlashDecoding++ — Faster Large Language Model Inference on GPUs	2023	Decode-side attention, where the batch is small and the kernel is bandwidth-bound.
Attention kernels inside a serving engine
FlashInfer — Efficient and Customizable Attention Engine for LLM Inference Serving	2025	A customizable attention engine underneath the serving stacks — where the kernel layer and the scheduler layer meet.
ragged paged attention — A High-Performance and Flexible LLM Inference Kernel for TPU	2026	The same paged-KV problem on TPU, where there are no warps and the memory system differs. A good forcing function for separating the algorithm from CUDA-specific folklore.
Triton kernel anatomy — The Anatomy of a Triton Attention Kernel	2025	Walks a paged-attention kernel from naive Triton to competitive with hand-tuned CUDA, naming each transformation and its payoff. The most assignment-adjacent reading in this section.
Triton-distributed — Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler	2025	Pushes communication primitives into the Triton kernel so compute and communication overlap at tile granularity rather than stream level — relevant when the all-reduce sits on the decode critical path.
Compilers and superoptimizers
TVM — An Automated End-to-End Optimizing Compiler for Deep Learning	2018	Compiling tensor programs to hardware: the schedule/compute split that Triton and TileLang later reuse.
Ansor — Generating High-Performance Tensor Programs for Deep Learning	2020	Search-based schedule generation — the auto-tuning half of the compiler story.
cuDNN — Efficient Primitives for Deep Learning	2014	The vendor-library layer that hand-written kernels compete against.
Hidet — Task-Mapping Programming Paradigm for Deep Learning Tensor Programs	2022	Pushes scheduling into the programming model instead of the search space.
Mirage — A Multi-Level Superoptimizer for Tensor Programs	2024	Superoptimization across kernel, thread-block, and thread levels at once.
MPK — A Compiler and Runtime for Mega-Kernelizing Tensor Programs	2025	Compiles a whole multi-GPU model into one persistent megakernel whose in-kernel scheduler dispatches SM-level tasks, eliminating per-op launch boundaries and enabling cross-operator pipelining and compute/communication overlap; the cost is that the runtime must carve SMs into workers and schedulers and size the task graph to the physical SM count.

Oct 5 Efficient LLM serving: paging, batching, and scheduling I

Prefill is compute-bound and batches well; decode is memory-bound and does not. Orca’s answer, iteration-level scheduling, is the baseline every later serving paper is measured against, so read it for the mechanism and for how cleanly the result is isolated. The supplementary entries are about predicting how long a request will run, which is what makes any of this schedulable in the first place. The allocator half of this meeting — paging the KV cache so that a scheduler has sequences to batch in the first place — is folded in below.

Paper	Year	Why read it
Iteration-level batching
Orca — A Distributed Serving System for Transformer-Based Generative Models	2022	Continuous (iteration-level) batching: let requests join and leave a batch every token instead of every request. Also a model of a well-argued systems evaluation.
NanoFlow — Towards Optimal Large Language Model Serving Throughput	2024	Intra-device parallelism: overlap compute, memory, and network within a single device to approach a throughput bound.
LoongServe — Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism	2024	Elastic sequence parallelism — vary the parallelism degree with the request, not with the deployment.
How long will this request run?
length prediction — Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction	2024	Scheduling wants the output length, which is exactly what is unknown. Predict it with a cheap proxy model.
response length perception — Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline	2023	Ask the model itself how long its answer will be, then schedule on that.
Scheduling for SLOs and fairness
VTC fairness — Fairness in Serving Large Language Models	2023	What fairness even means when requests cost wildly different amounts — and a scheduler that achieves it.
Niyama — Breaking the Silos of LLM Inference Serving	2025	Argues that one fleet per SLO class wastes capacity, and does fine-grained QoS-aware scheduling in a single shared pool; the tradeoff is that mixing classes makes tail-latency isolation an active scheduling problem.
SLOs-Serve — Optimized Serving of Multi-SLO LLMs	2025	Formulates per-request token allocation for multi-stage requests carrying application- and stage-specific SLOs, searching the chunked-prefill and optional speculative-decoding design space with a dynamic program. The objective is SLO-attaining serving capacity rather than raw throughput.
JITServe — SLO-aware LLM Serving with Imprecise Request Information	2025	Scheduling when output-length predictions are wrong, which is the realistic case. Pairs directly with the two length-prediction entries above by asking what the scheduler should do with a bad estimate rather than how to improve it.
locality-aware fair scheduling — Locality-aware Fair Scheduling in LLM Serving	2025	Shows VTC-style fairness and prefix-cache locality actively fight each other, and schedules against both. The natural follow-up to the VTC entry.
online scheduling with KV constraints — Online Scheduling for LLM Inference with KV Cache Constraints	2025	Treats KV-cache capacity as a hard online-algorithms constraint and gives competitive-ratio results — worth one session to see what provable structure sits underneath the heuristics everyone ships.

LLM serving basics, folded: paging and the KV allocator

Folded into the Oct 5 meeting above. This opened Part II on an earlier version of the schedule; it is now the front half of the batching meeting, and its lecture notes stand on their own. Two ideas carry most of the weight: manage KV memory in pages rather than contiguously, and batch at the granularity of a token rather than a request. Nearly every later paper assumes both, so read vLLM closely enough to explain why fragmentation was the binding constraint. The supplementary entries are the offload and preemption alternatives that lost.

Lecture notes for this meeting →

Paper	Year	Why read it
Paging the KV cache
vLLM / PagedAttention — Efficient Memory Management for Large Language Model Serving with PagedAttention	2023	Virtual memory for the KV cache. The single most important systems paper on the list — fragmentation, not compute, was the binding constraint.
vLLM docs — vLLM documentation	—	The serving stack most projects will build on.
Jenga — Effective Memory Management for Serving LLM with Heterogeneity	2025	SOSP’25, built on vLLM and co-authored by several of its core authors: PagedAttention’s uniform page assumption breaks on layers whose per-token KV footprints differ, such as sliding-window attention (Gemma-2) and hybrid Mamba/attention stacks (Jamba). Jenga’s two-level allocator generalizes to heterogeneous embedding sizes — the tradeoff is a more complex allocator for much less waste.
Offloading and preemption
FlexGen — High-Throughput Generative Inference of Large Language Models with a Single GPU	2023	Throughput-first offloading to CPU and disk. The opposite end of the latency spectrum from everything else here.
FastServe — Fast Distributed Inference Serving for Large Language Models	2023	Preemptive scheduling with multi-level feedback queues, borrowed straight from OS scheduling. The arXiv preprint predates the venue version by three years: it appeared at NSDI ’26 as “FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference.”
DeepSpeed-FastGen — High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference	2024	Dynamic SplitFuse — another take on mixing prefill and decode in one batch.
Engines and unusual targets
Pie — A Programmable Serving System for Emerging LLM Applications	2025	SOSP’25: breaks the monolithic generation loop into service handlers that user “inferlets” drive, so applications supply their own KV-cache and generation policy — read it against vLLM to see what the engine gives up in global scheduling authority when it exposes that control.
WaferLLM — Large Language Model Inference at Wafer Scale	2025	OSDI’25: rederives LLM parallelism for a mesh of hundreds of thousands of cores with no HBM, showing which GPU-era assumptions (memory-bound decode, all-reduce collectives) are architectural rather than fundamental.
PrefillOnly — An Inference Engine for Prefill-only Workloads in Large Language Model Applications	2025	For discriminative requests that emit exactly one token (recommendation, credit verification, data labeling), the engine stores only the last computed layer’s KV cache and can predict each job’s completion time before it starts, which enables shortest-remaining-job-first scheduling. Shows what the engine looks like when you delete the decode phase.
Measurement and surveys
LLM inference characterization — A Systematic Characterization of LLM Inference on GPUs	2025	Measurement study that splits inference into prefill and decode and traces each phase’s cost to kernel mix, roofline bounds, issue stalls, and memory access patterns, on an A100 server plus a Jetson Orin edge device — the empirical grounding for the roofline claims the rest of the course makes.
Taming the Titans — A Survey of Efficient LLM Inference Serving	2025	Current taxonomy of the serving stack (instance-level, cluster-level, emerging scenarios); useful as the map to read once before the specialized weeks.

Oct 7 Efficient LLM serving: batching and scheduling II

If prefill and decode want different hardware and different batching, you can either interleave them carefully on one machine or split them across machines and ship the KV cache between them. Sarathi-Serve is the first answer; the disaggregation topics are the second, and they disagree about what the split costs. Watch for what lands on the critical path: a large, latency-sensitive transfer that did not exist before.

Paper	Year	Why read it
Interleaving both phases on one machine
Sarathi-Serve — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve	2024	Chunked prefill plus stall-free batching: split long prefills so decodes are never blocked behind them.
TaiChi — Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving	2025	The framing paper for this meeting: aggregation wins under tight TTFT, disaggregation under tight TPOT, and neither is optimal when both are constrained.
Splitting prefill from decode
DistServe — Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving	2024	The clean statement of the idea: disaggregate the two phases, optimize goodput under both TTFT and TPOT constraints.
Splitwise — Efficient generative LLM inference using phase splitting also Nov 23	2023	The same split, argued from power and hardware heterogeneity — and backed by a production trace.
P/D-Serve — Serving Disaggregated Large Language Model at Scale	2024	Disaggregation as actually operated at scale, including the parts that are unglamorous.
Arrow — Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture	2025	Flips instances between prefill and decode roles as the request mix drifts, attacking the static-ratio provisioning problem every fixed P/D deployment has.
Moving the KV cache between them
Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving also Oct 21, Nov 4, Nov 23	2024	A KVCache-centric architecture in production at scale: disaggregation plus a distributed cache pool plus overload handling.
MemServe — Context Caching for Disaggregated LLM Serving with Elastic Memory Pool also Nov 4	2024	An elastic memory pool that makes cached context a first-class, shareable resource across instances.
CloudMatrix384 — Serving Large Language Models on Huawei CloudMatrix384	2025	A full-stack industrial account of disaggregated serving on a non-NVIDIA supernode with a peer-to-peer unified bus — a useful data point on what changes when all-to-all bandwidth stops being scarce.
Sharing one GPU between the phases
semi-PD — Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage	2025	Disaggregates compute between phases at the SM level while keeping one shared KV store on the same GPU, buying phase isolation without the cross-GPU KV transfer DistServe and Splitwise pay.
Adrenaline — Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation	2025	Splits at a finer seam than prefill/decode by offloading the memory-bound part of decode attention onto the prefill instances, whose memory capacity and bandwidth sit idle while they run compute-bound work. The P/D boundary is not the only place to cut.
Nexus — Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving	2025	Does the P/D split inside a single GPU via SM partitioning, for the regime where separate fleets are unaffordable. Read against Sarathi-Serve’s chunked-prefill answer to the same interference problem.

Oct 14 Efficient LLM serving: routing and load balancing

Above a single replica: which instance should serve this request, what happens when the fleet goes imbalanced, and how to share GPUs across many models or adapters. Llumnix is the migration argument — read it for why rescheduling a live request is worth the copy. The mixture-of-experts entries are here because sparse activation hands the same load-balancing problem to the layer below: the weights you need depend on the token.

Paper	Year	Why read it
Routing and live migration
Llumnix — Dynamic Scheduling for Large Language Model Serving	2024	Live migration of running requests between instances — scheduling as a continuous, not one-shot, decision.
Preble — Efficient Distributed Prompt Scheduling for LLM Serving also Nov 4	2024	Prefix sharing across a distributed cluster, which turns cache reuse into a scheduling and placement problem.
DualMap — Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving	2026	Quantifies the cache-affinity versus load-balance conflict, then hashes each request to two candidate instances via independent prompt hashes and picks by live load. Power-of-two-choices adapted to prefix locality, with a fallback to load-aware placement once TTFT breaches the SLO.
RouteBalance — Fused Model Routing and Load Balancing for Heterogeneous LLM Serving	2026	Joins two decisions usually made separately — which model answers, and which instance runs it — into one online assignment that prices latency at model-selection time, using batched predictors and dead-reckoned instance state cheap enough to run synchronously on the hot path.
Multiplexing many models on a cluster
AlpaServe — Statistical Multiplexing with Model Parallelism for Deep Learning Serving	2023	Statistical multiplexing with model parallelism: parallelism is a latency tool under bursty load, not just a capacity tool.
MuxServe — Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving	2024	Spatial-temporal multiplexing of several LLMs on shared GPUs.
DynamoLLM — Designing LLM Inference Clusters for Performance and Energy Efficiency	2024	Energy as a first-class objective for an inference cluster.
Clipper — A Low-Latency Online Prediction Serving System	2016	Pre-LLM prediction serving. Worth reading for how much of the modern stack is a rediscovery, and how much genuinely is not.
ServerlessLLM — Low-Latency Serverless Inference for Large Language Models	2024	Cold start is the whole problem when models are gigabytes; make checkpoint loading fast.
Serving many adapters
S-LoRA — Serving Thousands of Concurrent LoRA Adapters	2023	Thousands of adapters over one base model: unified paging for both KV cache and adapter weights.
Punica — Multi-Tenant LoRA Serving	2023	A batched kernel that lets requests using different adapters share one batch.
LoRAServe — Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems	2025	Handles rank heterogeneity across adapters with workload-aware dynamic placement and remote adapter access over GPU Direct RDMA. Pushes multi-LoRA serving past the node-local loading, caching, and kernel work it builds on, to where adapters get rebalanced across GPUs under workload drift.
Mixture of experts: the routing idea
GShard — Scaling Giant Models with Conditional Computation and Automatic Sharding	2020	Conditional computation with automatic sharding — where the modern MoE recipe starts.
Switch Transformer — Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	2021	Top-1 routing, and a careful account of the instability that comes with it.
Mixtral — of Experts	2024	An open-weight sparse MoE, which is what makes MoE serving experiments feasible for a course project.
DeepSeekMoE — Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models	2024	Fine-grained experts plus shared experts — the architecture behind DeepSeek-V3.
expert choice routing — Mixture-of-Experts with Expert Choice Routing	2022	Invert the assignment: experts pick tokens, which fixes load balance by construction.
Mixture of experts: the systems problem
MegaBlocks — Efficient Sparse Training with Mixture-of-Experts	2022	Block-sparse kernels that remove the token-dropping hack entirely.
Tutel — Adaptive Mixture-of-Experts at Scale	2022	Adaptive parallelism and pipelining for MoE, switchable at runtime.
DeepSpeed-MoE — Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale	2022	MoE inference as a systems problem: expert placement, communication, and distillation.
Fiddler — CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models	2024	Run some experts on the CPU when they will not fit on the GPU.
MoE-Infinity — Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache	2024	Expert activation is skewed and repetitive, so cache experts the way you would cache anything else.
pre-gated MoE — An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference	2023	Decide the routing one layer early so the expert weights can be prefetched.
MegaScale-Infer — Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism	2025	The MoE serving system this section was missing: because MoE makes FFNs memory-bound at decode, it disaggregates attention from FFN and scales each independently. The cost is a communication library you now own.
Expert-as-a-Service — Towards Efficient, Scalable, and Robust Large-scale MoE Serving	2025	Turns experts into independent, stateless services rather than ranks in a fixed expert-parallel group, so partial hardware failures degrade throughput instead of halting the deployment and per-expert capacity can track live traffic. The cost is a CPU-free peer-to-peer communication layer underneath.
Step-3 — Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding	2025	Designs the attention and MoE structure backwards from decoding cost on specific hardware. The strongest recent example of model architecture as a serving decision.

Oct 19 Efficient LLM serving: KV-cache optimization I

The KV cache grows linearly with context and batch size, and it is read in full for every token generated — so it is simultaneously the capacity limit and the bandwidth limit. This first meeting takes the capacity side: make the cache smaller before anything has to read it. Start from KIVI: 2-bit KV quantization is the largest cut available without touching the model, and its asymmetric treatment of keys and values is the kind of detail that decides whether a compression scheme survives contact with a real model. The eviction topic throws tokens away rather than compressing them, and MQA and GQA are here because they change the size of the problem before any system touches it. Where the cache lives, and how much of it a decode step must read, is the next meeting.

Paper	Year	Why read it
Quantizing the cache
KIVI — A Tuning-Free Asymmetric 2bit Quantization for KV Cache	2024	2-bit KV quantization, asymmetric by axis: per-channel for keys, per-token for values. Tuning-free, which is why it is deployable.
KVQuant — Towards 10 Million Context Length LLM Inference with KV Cache Quantization	2024	Pushes KV quantization far enough that context length, not memory, becomes the limit again.
Oaken — Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization	2025	Splits KV quantization into an offline pass that fixes outlier thresholds and an online pass that only sets the scale, so online outlier detection stops eating the savings; custom quantization engines and memory-management units carry it into an accelerator. Read as the algorithm-hardware co-design answer to GPU-side KV quantization like KVQuant and KIVI.
Storing fewer keys and values
MQA — Fast Transformer Decoding: One Write-Head is All You Need	2019	Multi-query attention: one KV head instead of many. The single largest constant-factor cut to KV cache size.
GQA — Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints	2023	The compromise between multi-head and multi-query that nearly every current open model ships.
Kimi Linear — An Expressive, Efficient Attention Architecture	2025	A hybrid linear-attention and MLA stack that shrinks the KV cache by changing the architecture. The escape hatch to contrast with compress-and-evict approaches.
Evicting tokens
SnapKV — LLM Knows What You are Looking for Before Generation	2024	The attention pattern over the prompt predicts which entries matter, before generation starts.
H2O — Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models	2023	Heavy hitters: a small set of tokens receives most of the attention mass, so the rest can be evicted.
StreamingLLM — Efficient Streaming Language Models with Attention Sinks	2023	Attention sinks — keeping the first few tokens is what makes a sliding window not collapse. A genuinely surprising empirical result.
Scissorhands — Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time	2023	Importance persists across steps, so an eviction decision made once stays roughly right.
FastGen — Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs	2023	Different attention heads deserve different eviction policies; profile them and specialize.
PyramidKV — Dynamic KV Cache Compression based on Pyramidal Information Funneling	2024	Budget the cache by layer rather than uniformly — deeper layers need less.
Moving and accounting for the cache
CacheGen — KV Cache Compression and Streaming for Fast Large Language Model Serving also Nov 23	2023	If cached KV has to cross a network, its encoding is a bandwidth problem, not just a memory problem.
vToken — Token-Level Virtualization for Reclaimable KV Caches	2026	Token-table indirection decouples logical token liveness from physical block placement, so the intra-block fragmentation left by token-granularity eviction can actually be reclaimed by asynchronously repacking live tokens — while keeping PagedAttention kernels and CUDA Graphs intact.
KV cache management survey — From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving also Nov 23	2026	From a tensor buffer to a distributed memory hierarchy — the survey to read before picking a presentation paper in this area.

Oct 21 Efficient LLM serving: KV-cache optimization II

The same cost attacked from the other two directions: read less of the cache per token, and stop assuming all of it belongs in HBM. Quest is the algorithm side — page-level criticality estimation that leaves the attention output nearly intact while touching a fraction of the pages — and Mooncake is the systems side, a KV-cache-centric architecture that pools and tiers the cache across a cluster rather than treating it as per-GPU scratch. Read them against each other: one changes what attention reads, the other changes where the bytes are. The remaining topics offload the cache, predict it, and sparsify it, and cover the sparse-attention literature that pair stands on. Compression and eviction were the previous meeting.

Paper	Year	Why read it
A KV cache tier across the cluster
Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving also Oct 7, Nov 4, Nov 23	2024	A KVCache-centric architecture in production at scale: disaggregation plus a distributed cache pool plus overload handling.
AttentionStore — Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention	2024	A cache hierarchy for multi-turn conversations, where the reuse is across turns and sessions.
prediction-based KV management — Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management also Nov 23	2026	Predict what a dynamic agent workflow will need next and manage the cache accordingly.
Reading less of the cache per token
Quest — Query-Aware Sparsity for Efficient Long-Context LLM Inference	2024	Query-aware page selection: pick which KV pages to attend to per query, per layer.
InfiniGen — Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management	2024	Speculate which KV entries the next layer will actually attend to, and prefetch only those from CPU memory.
HiSparse — Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management	2026	Hierarchical KV cache management for sparse-attention decoding.
ShadowKV — KV Cache in Shadows for High-Throughput Long-Context LLM Inference	2024	Keep a low-rank key cache on GPU and offload values, reconstructing on demand.
DuoAttention — Efficient Long-Context LLM Inference with Retrieval and Streaming Heads	2024	Only some heads need the full context; give the others a streaming window.
LServe — Efficient Long-sequence LLM Serving with Unified Sparse Attention	2025	Unifies static and dynamic sparsity in one serving system rather than one kernel.
XAttention — Block Sparse Attention with Antidiagonal Scoring	2025	Estimates block importance from antidiagonal sums, making block selection cheap enough not to eat the sparsity savings — the recurring failure mode in this literature.
Star Attention — Efficient LLM Inference over Long Sequences	2024	Two-phase block-sparse attention aimed at distributed long-context inference.
Sparsity built into the model
NSA — Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention	2025	Sparse attention designed to be trainable and hardware-aligned, rather than bolted on at inference time.
MoBA — Mixture of Block Attention for Long-Context LLMs	2025	Block-level routing of attention, in the spirit of mixture-of-experts.
DeepSeek-V3.2 — Pushing the Frontier of Open Large Language Models	2025	Sparse attention trained into the model rather than bolted on at inference time, at frontier scale: DeepSeek Sparse Attention cuts long-context attention complexity while the model still holds up on reasoning and agentic tool use. Read against NSA and MoBA as the natively-trained end of the sparse-attention spectrum.
sparse frontier — The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs	2025	Controlled evaluation of six training-free sparse attention methods across lengths, model sizes, and sparsity levels, finding that neither prefill selection strategy dominates and the optimal choice is task-dependent. The corrective to read against Quest and MInference.
Long context across devices
ring attention — with Blockwise Transformers for Near-Infinite Context	2023	Distribute a long sequence across devices and overlap the KV transfer with compute.
MInference — 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention	2024	Long-prompt prefill is the bottleneck in RAG and agent workloads; exploit the structure in the attention pattern.

Oct 28 Student sharing — what you optimized and what it bought (Assignment 3)

Nothing to read for this meeting. Bring your Assignment 3 numbers — the lever you expected to win, the one that actually did, and the gap between them.

Nov 4 Efficient LLM serving: prefix cache

Agent and chat workloads send prompts that share long prefixes: a system prompt, a document, the conversation so far. Recomputing that prefix is pure waste, and the first two topics are the two halves of the fix — how to index shared prefixes, and how to schedule requests so the sharing actually pays. Everything here is cheap to compute and expensive to store, so the interesting question is what to evict — a natural final project question, and the one an earlier version of this course ran as a competition. The eviction-policy background is in Background: Caching and Eviction.

Lecture notes for this meeting →

Paper	Year	Why read it
Indexing shared prefixes
SGLang / RadixAttention — SGLang: Efficient Execution of Structured Language Model Programs also Sep 16, Nov 18	2023	A radix tree over cached prefixes with LRU eviction, plus a language whose structure exposes the sharing.
ChunkAttention — Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition	2024	A prefix-aware KV structure with a two-phase kernel, so sharing does not cost attention throughput.
Prompt Cache — Modular Attention Reuse for Low-Latency Inference	2023	Reuse at the level of declared, reusable prompt modules rather than raw token prefixes.
Hydragen — High-Throughput LLM Inference with Shared Prefixes	2024	Decompose attention into shared-prefix and per-sequence parts so the shared half becomes a dense matmul.
Scheduling so the sharing pays
Preble — Efficient Distributed Prompt Scheduling for LLM Serving also Oct 14	2024	Prefix sharing across a distributed cluster, which turns cache reuse into a scheduling and placement problem.
Parrot — Efficient Serving of LLM-based Applications with Semantic Variable also Sep 16, Nov 18	2024	Semantic Variables expose the dependency structure of a multi-call application to the scheduler, so it can optimize end-to-end latency instead of per-request latency.
Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving also Oct 7, Oct 21, Nov 23	2024	A KVCache-centric architecture in production at scale: disaggregation plus a distributed cache pool plus overload handling.
MemServe — Context Caching for Disaggregated LLM Serving with Elastic Memory Pool also Oct 7	2024	An elastic memory pool that makes cached context a first-class, shareable resource across instances.
Prefix caching for RAG
CacheBlend — Fast Large Language Model Serving for RAG with Cached Knowledge Fusion	2024	RAG chunks share no common prefix. Recompute selectively so non-prefix reuse stays correct.
RAGCache — Efficient Knowledge Caching for Retrieval-Augmented Generation	2024	A multilevel cache of retrieved-document KV state, informed by how skewed retrieval actually is.
Cache-Craft — Managing Chunk-Caches for Efficient Retrieval-Augmented Generation	2025	RAG chunks recur across queries but land at arbitrary positions, so prefix caching misses them; this caches per-chunk KV and does selective recomputation to repair a reused cache instead of recomputing the whole thing. Read alongside CacheBlend for the accuracy-versus-recompute knob.
IC-Cache — Efficient Large Language Model Serving via In-context Caching	2025	Exploits requests that are semantically similar rather than textually prefix-identical, replaying past request-response pairs as in-context examples so smaller models can absorb work that would otherwise need a larger one. A genuinely different reuse axis from exact-token caching.
Cache pools and hybrid models
LMCache — An Efficient KV Cache Layer for Enterprise-Scale LLM Inference	2025	The KV cache layer that plugs under both vLLM and SGLang to share caches across engines and queries, spanning GPU, CPU, storage, and network with pipelined data movement and PD transfer. Worth reading for what enterprise deployment breaks — notably that context truncation halves the prefix cache hit rate.
TokenLake — A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving	2025	Pools prefix cache at segment rather than instance granularity behind a declarative cache interface, with heavy-hitter-aware load balancing, so hot prefixes stop pinning requests to one replica. Read against cache-aware routing and cache-centric PD-disaggregation, which are its two baselines.
Marconi — Prefix Caching for the Era of Hybrid LLMs	2024	Prefix caching when the model is a hybrid (attention plus SSM) and cached state is no longer a simple per-token array.

Nov 9 Efficient LLM serving: pruning and quantization

One meeting covering both halves of the topic: the block below is the outlier story first, the post-training line second. Fewer bits per weight or per KV entry buys memory and bandwidth, and the whole game is where the error goes. LLM.int8() is the outlier paper: read it for why naïve INT8 fails on large models specifically. The pruning entries are here for contrast — quantization shrinks every weight, pruning removes weights outright, and only structured sparsity is something the hardware can actually exploit.

Lecture notes for this meeting →

Paper	Year	Why read it
Outliers, and why naïve INT8 fails
LLM.int8() — 8-bit Matrix Multiplication for Transformers at Scale	2022	The outlier problem, discovered: a few feature dimensions have huge magnitudes and wreck naive INT8. Mixed-precision decomposition as the fix.
SmoothQuant — Accurate and Efficient Post-Training Quantization for Large Language Models	2022	Migrate quantization difficulty from activations to weights with a per-channel rescale.
ZeroQuant — Efficient and Affordable Post-Training Quantization for Large-Scale Transformers	2022	An early end-to-end PTQ pipeline with fused kernels; useful as a baseline.
Number formats
mixed precision — Training	2017	The original FP16 training recipe: loss scaling and an FP32 master copy.
FP8 formats — for Deep Learning	2022	Why E4M3 and E5M2 exist and where each is used.
INT vs FP — INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats	2025	Controlled comparison of integer against floating-point formats at matched block granularity, showing the winner flips with block size and bit width. Read before assuming FP4 beats INT4 by construction.
microscaling FP4 — Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization	2025	Diagnoses why MXFP4’s power-of-two block scales lose accuracy that NVFP4’s retain, and what it takes to close the gap. Makes the format’s numerics concrete rather than a vendor spec sheet.
Pruning, sparsity, and cheaper architectures
SparseGPT — Massive Language Models Can Be Accurately Pruned in One-Shot	2023	One-shot pruning to 50% sparsity without retraining, using the same second-order machinery as GPTQ.
Wanda — A Simple and Effective Pruning Approach for Large Language Models	2023	Weights times input activations, and nothing else. The baseline any pruning method has to beat.
LLM-Pruner — On the Structural Pruning of Large Language Models	2023	Structural pruning, which is the kind that actually makes a GPU faster.
ProxSparse — Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs	2025	Learns semi-structured sparsity masks by regularized optimization that turns non-differentiable mask selection into a smooth search, instead of the local layer-wise heuristics one-shot criteria use, and applies no weight update once the mask is fixed. The learning-based end of the mask-selection design space, evaluated across seven models.
beyond FLOPs — Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy	2026	Reorganizes pruning methods by which GEMM dimension (M, N, or K) they actually shrink, then measures realized speedup rather than FLOP reduction under one implementation-consistent harness. Finds static depth pruning is the strongest Pareto-optimal baseline at low quality loss, with the frontier shifting to dynamic depth and then static width as the loss budget grows.
Nemotron-H — A Family of Accurate and Efficient Hybrid Mamba-Transformer Models	2025	Production prune-and-distill on a hybrid Mamba-Transformer backbone: MiniPuzzle compresses the 56B model into a 47B with comparable accuracy that infers 20% faster, on top of an architecture that already swaps most attention for constant-per-token Mamba state. Structured pruning as a shipping step rather than a research demo.

Efficient LLM serving: pruning and quantization, continued: the post-training line

Folded into the Nov 9 meeting above. The post-training quantization line, and the rotation trick that dodges outliers instead of special-casing them. Read GPTQ for the method and QServe for what it costs to serve a quantized model at speed — the gap between a good compression ratio and a good token rate is the whole point of pairing them. The supplementary entries are the extreme end: 4-bit, 2-bit, and ternary.

Paper	Year	Why read it
Post-training weight quantization
GPTQ — Accurate Post-Training Quantization for Generative Pre-trained Transformers	2022	One-shot post-training quantization with second-order error compensation, layer by layer.
AWQ — Activation-aware Weight Quantization for LLM Compression and Acceleration	2023	Not all weights matter equally; protect the salient 1% using activation statistics.
OmniQuant — Omnidirectionally Calibrated Quantization for Large Language Models	2023	Learn the clipping ranges and equivalent transforms rather than hand-tuning them.
SqueezeLLM — Dense-and-Sparse Quantization	2023	Dense low-bit weights plus a sparse outlier set kept at full precision.
Atom — Low-bit Quantization for Efficient and Accurate LLM Serving	2023	Low-bit quantization evaluated as a serving system, in end-to-end throughput rather than perplexity.
Rotations instead of special cases
QuIP# — Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks	2024	Hadamard incoherence plus lattice codebooks — the strong 2-bit result.
QuaRot — Outlier-Free 4-Bit Inference in Rotated LLMs	2024	Rotate the model so outliers cannot exist in any basis, then quantize everything to 4 bits.
SpinQuant — LLM quantization with learned rotations	2024	Learn the rotation instead of fixing it.
block rotation MXFP4 — Block Rotation is All You Need for MXFP4 Quantization	2025	Benchmarks PTQ methods under MXFP4 and finds the global rotations of QuaRot and SpinQuant collapse there, tracing the cause to MXFP4’s power-of-two per-block scales rather than to rotation itself. Matching rotation granularity to the 32-element block restores the gains — a direct correction to reading SpinQuant-style rotation as format-agnostic.
Serving a quantized model at speed
QServe — W4A8KV4 Quantization and System Co-design for Efficient LLM Serving	2024	W4A8KV4 with the system co-designed alongside the algorithm — the paper that takes dequantization overhead seriously.
SageAttention3 — Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training	2025	Quantizes the attention matmuls themselves rather than only the KV cache, using FP4 microscaling on Blackwell tensor cores plus a two-level scheme for the post-softmax probability matrix, whose narrow [0,1] range is the hard case. The 8-bit training variant is lossless for fine-tuning but converges more slowly in pretraining.
Tilus — A Tile-Level GPGPU Programming Language for Low-Precision Computation	2025	Makes arbitrary low bit widths first-class with a layout system for register distribution, so W3A16 or W5A8 stops requiring a bespoke hand-written kernel. The systems answer to why odd bit widths never ship.
The extreme end: 4-bit, 2-bit, ternary
BitNet b1.58 — The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits	2024	Ternary weights trained from scratch. Read it for the claim that quantization belongs in training, not after.
BitNet b1.58 2B4T — BitNet b1.58 2B4T Technical Report	2025	The first open-source native 1.58-bit model at 2B parameters and 4T tokens, reported on par with full-precision open models of similar size. Ships its own GPU and CPU inference implementations, so the memory, energy, and decoding-latency claims for ternary weights come with a runnable stack rather than a format proposal.
ParetoQ — Improving Scaling Laws in Extremely Low-bit LLM Quantization	2025	A unified training recipe that makes 1, 1.58, 2, 3, and 4-bit directly comparable, locating the sharp representational break between 2 and 3 bits. Turns “how low can you go” from anecdote into a Pareto curve.
Quartet — Native FP4 Training Can Be Optimal for Large Language Models	2025	Fits a low-precision scaling law across bit widths and training configurations, picks the accuracy-per-unit-compute optimum from it, and ships Blackwell CUDA kernels that keep every linear layer natively in FP4 with no mixed-precision fallback, reporting training competitive with FP8 and FP16. A template for treating precision as a systems decision.
NVFP4 pretraining — Pretraining Large Language Models with NVFP4	2025	A large FP4 pretraining run with the stabilizers it required — Hadamard transforms, stochastic rounding, selective high-precision layers. Shows which layers refuse to go low-precision even at scale.

Nov 11 Efficient LLM serving: speculative decoding

Decode is memory-bound, so verifying several tokens costs barely more than generating one: guess ahead cheaply, then verify in parallel. Speculative decoding and EAGLE sit at opposite ends of the design space — an independent draft model versus a head trained on the target model’s own features. The variants differ mainly in where guesses come from and how they are verified; read two or three and the pattern is clear.

Paper	Year	Why read it
Guess ahead, verify in parallel
speculative decoding — Fast Inference from Transformers via Speculative Decoding	2022	The original formulation, with the rejection-sampling argument for why the output distribution is preserved exactly.
speculative sampling — Accelerating Large Language Model Decoding with Speculative Sampling	2023	The concurrent DeepMind version; read alongside the above for the same idea argued differently.
blockwise parallel decoding — for Deep Autoregressive Models	2018	The 2018 ancestor of the whole idea.
Where the guesses come from
EAGLE — Speculative Sampling Requires Rethinking Feature Uncertainty	2024	Draft in feature space rather than token space, one autoregressive head on top of the target model.
EAGLE-2 — Faster Inference of Language Models with Dynamic Draft Trees	2024	Context-dependent dynamic draft trees.
EAGLE-3 — Scaling up Inference Acceleration of Large Language Models via Training-Time Test	2025	The EAGLE variant every major engine ships (vLLM, SGLang, TensorRT-LLM): it drops EAGLE’s feature-prediction constraint for direct token prediction plus multi-layer feature fusion, so draft quality keeps improving with training data instead of saturating.
Medusa — Simple LLM Inference Acceleration Framework with Multiple Decoding Heads	2024	Multiple decoding heads instead of a separate draft model — no second model to serve.
self-speculative — Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding	2023	Draft by skipping layers of the model itself.
REST — Retrieval-Based Speculative Decoding	2023	Retrieve draft continuations from a corpus instead of generating them.
lookahead decoding — Break the Sequential Dependency of LLM Inference Using Lookahead Decoding	2024	Draft with Jacobi iteration — no draft model and no training at all.
Verification trees, long context, and scale
Sequoia — Scalable, Robust, and Hardware-aware Speculative Decoding	2024	Treats the draft tree shape as an optimization problem, hardware-aware.
SpecInfer — Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification	2023	Tree-based speculation with a token-tree verifier, framed as a serving system.
SpecExec — Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices	2024	Very large speculative trees when the target model is offloaded and each step is expensive.
MagicDec — Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding	2024	Speculative decoding for long context and large batch, where the usual assumptions invert.
mirror speculative decoding — Breaking the Serial Barrier in LLM Inference	2025	Breaks draft-then-verify serialization by running both directions at once — the draft proposes continuations while the target proposes correction paths for the draft — spread across heterogeneous accelerators. The tradeoff is a deployment that needs both a GPU and an NPU plus a tight cross-device synchronization budget.
Inside a serving system
online spec decoding — Online Speculative Decoding	2023	Keep updating the draft model on the live query distribution.
AdaServe — Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding	2025	Makes per-request speculation a scheduling variable: a constrained-optimization formulation builds a speculation tree sized to each request’s latency target, and a speculate-select-verify pipeline trades decode speed against throughput within one shared batch. The serving-system view of speculative decoding once SLOs stop being uniform.
SpecForge — A Flexible and Efficient Open-Source Training Framework for Speculative Decoding	2026	The SGLang team on actually training draft models for production targets: target-draft decoupling, hybrid parallelism, and tuned training kernels, plus SpecBundle, a set of released EAGLE-3 drafts for mainstream open models. The practical blocker it addresses is that good draft models, not the algorithm, are what the community is missing.
spec decoding meets quantization — Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design	2025	Shows the two optimizations interfere: quantization moves the memory-bound to compute-bound crossover earlier, shrinking speculation’s headroom. A rare paper about composing optimizations rather than one in isolation.
Surveys
spec decoding survey — Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding	2024	Read this before picking a presentation paper in this area.
spec decoding survey 2025 — Speculative Decoding and Beyond: An In-Depth Survey of Techniques	2025	Newer and broader than the 2024 speculative decoding survey: organizes draft-verify by generation strategy (n-gram through draft models) and by refinement mechanism (single-pass versus iterative verification), and extends past text into image and speech generation. Useful as the taxonomy backbone for this section.

Nov 18 Efficient agent serving systems I

Parrot returns, now from the serving side. An agent request is really a DAG of dependent model calls, and a system that can see the DAG can schedule it, deduplicate its prefixes, and stop optimizing each call in isolation. This is the premise of the two-lecture agent-serving block that closes Part II, so read it as a design proposal rather than as a set of numbers. The scheduling half of the meeting is the block below.

Lecture notes for this meeting →

Paper	Year	Why read it
Declared structure: the request as a program
Parrot — Efficient Serving of LLM-based Applications with Semantic Variable also Sep 16, Nov 4	2024	Semantic Variables expose the dependency structure of a multi-call application to the scheduler, so it can optimize end-to-end latency instead of per-request latency.
SGLang — Efficient Execution of Structured Language Model Programs also Sep 16, Nov 4	2023	A front-end language plus a runtime, co-designed. Read it here for the programming model; the RadixAttention half is in prefix caching.
Structure the program never declared
InferCept — Efficient Intercept Support for Augmented Large Language Model Inference also Sep 23, Nov 23	2024	What to do with GPU state while a request is blocked on a tool call: keep it, discard it, or swap it out.
Pythia — Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving	2026	Analyzes production traces from an agent-serving platform and an internal coding assistant — low prefix-cache hit rates, contention from long-context requests, queuing from poor scaling — then has the application expose workflow semantics through a small serving-layer interface the scheduler can exploit. Read it against Parrot for how thin that declaration interface can get.
Serving layers for agentic applications
agentic batch query optimization — Batch Query Processing and Optimization for Agentic Workflows	2025	Halo compiles each agent workflow into a query-plan DAG and consolidates concurrent workflows into one graph so shared computation runs once, with a cost model over prefill/decode cost, cache reuse, and GPU placement driving plan-level optimization. The cleanest statement of the database view of agent serving.
workflow-aware serving — A Workflow-Aware Serving Layer for Agentic Applications	2026	Dyserve sits between agent frameworks and serving engines, solving one ILP per workflow that picks each DAG node’s model, verifier, and backend over a heterogeneous pool, pre-solved at several pressure levels at admission so load shifts can redirect a workflow’s uncommitted suffix without re-running the solver. Read it for how much structure the app must expose to make that solve possible.
software-defined agentic serving — Software-Defined Agentic Serving	2026	A short position paper arguing agentic serving should be programmable and system-aware rather than statically parameterized, sketching an SDN-inspired framework that adjusts communication attributes from runtime state. Worth reading for whether the SDN analogy holds when the “packets” carry gigabytes of KV state.

Efficient agent serving systems I, continued: scheduling agentic programs

Folded into the Nov 18 meeting above. Scheduling when the unit of work is a program rather than a request. The scheduler has to reason about dependencies between calls, about which program is closest to finishing, and about what is worth keeping in cache between steps — and it has to do it without the program telling it anything. Read Autellix and Teola against each other: one infers structure, the other asks the program to declare it.

Paper	Year	Why read it
Scheduling programs rather than requests
Autellix — An Efficient Serving Engine for LLM Agents as General Programs	2025	Treats an agent program, not a request, as the scheduling unit — and shows what head-of-line blocking costs when it is not. Published at NSDI ’26 under the name Agentix; search that title for the venue version.
SAGA — Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters	2026	Workflow-atomic scheduling: schedule the whole agent workflow as a unit on a GPU cluster.
Continuum — Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live also Nov 23	2025	Multi-turn agent scheduling with an explicit time-to-live on cached state: how long is a session’s KV cache worth keeping?
SMetric — Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling	2026	Argues cluster throughput, not per-token latency, is the right objective once agents rather than humans consume responses, and shows cache-aware routing overloads a few instances to protect KV reuse. It balances only each session’s first request and routes the follow-ups cache-aware — the tradeoff being reuse against load balance, not latency against fairness.
End-to-end pipeline optimization
Teola — Towards End-to-End Optimization of LLM-based Applications also Sep 23	2024	End-to-end optimization across the whole application dataflow rather than the model call alone.
Helium — Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective	2026	Serving agentic workflows from a data-systems angle.
HexAGenT — Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling	2026	Schedules agentic workflows as online-revealed DAGs across heterogeneous prefill-decode disaggregated clusters, ranking ready calls by projected risk of missing the workflow’s completion horizon and jointly picking prefill placement, decode placement, and queue priority under KV capacity and cross-stage transfer cost. The objective is workflow-level latency, not per-call latency.
What the workload looks like
agentic AI workloads — Agentic AI Workload Characteristics also Sep 23	2026	A characterization of agentic workloads — the trace-level view of what these systems actually receive.
observation not prediction — Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving	2026	Deliberately refuses to predict agent behavior and schedules on observed conversation state instead. A useful robustness argument against the prediction-based approaches that dominate this section.
Agentic RL as a serving workload
rollout infrastructure tax — The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning	2026	A measurement study of four execution substrates for coding-agent RL rollouts — single containers, hosted sandboxes, Kubernetes, and cloud VMs — finding up to 110x variation in cold-start latency and a 1.8x spread in projected rollout worker-hours at million-trajectory scale. Read the scope carefully: it models rollout workers only and explicitly excludes GPU inference and optimizer steps.
TideRL — Boosting Agentic RL Goodput with Readiness-Aware Scheduling	2026	Schedules RL rollouts by readiness so straggler trajectories stop idling the trainer — the same stall-hiding logic as agent serving, with on-policy staleness bounding how far you may reorder.
Libra — Efficient Resource Management for Agentic RL Post-Training	2026	Manages the shifting split between rollout and training resources during agentic RL, where long-tailed trajectories make a static partition badly wasteful.

Nov 23 Efficient agent serving systems II

An agent request stops generating whenever it calls a tool, and the tool has to run somewhere isolated. That creates two problems the classic serving stack never had: what to do with GPU state during a stall — hold it, discard it, or swap it — and how to start and checkpoint a sandbox fast enough that the sandbox is not the new bottleneck. Session state and multi-agent workloads are the two blocks that follow.

Paper	Year	Why read it
Tool stalls: what to do with the GPU state
InferCept — Efficient Intercept Support for Augmented Large Language Model Inference also Sep 23, Nov 18	2024	What to do with GPU state while a request is blocked on a tool call: keep it, discard it, or swap it out.
parallelizing tool execution — Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving	2026	Overlap tool execution with generation instead of serializing them.
MORI — Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI	2026	Ranks agent programs on a continuous idleness spectrum rather than a binary busy/idle label, putting the busiest in GPU HBM and the most idle in CPU DRAM and shifting the tier boundary to match the hardware’s capacity ratio. The tradeoff is cross-tier KV transfer cost against how well a relative idleness ranking predicts the next stall — the constructive counterpart to InferCept’s discard-or-retain choice at an interception.
speculative tool calls — Optimizing Agentic Language Model Inference via Speculative Tool Calls	2025	Speculates tool calls before the model commits to them and forces sequences to stay resident in the engine so a tool round trip does not cost a re-prefill, with a theoretical analysis of which speculation configurations pay off and a proposed “tool cache” endpoint for providers to expose. The question it leaves open is non-idempotent tools, where a wrong speculation has already had an effect.
Atomix — Timely, Transactional Tool Use for Reliable Agentic Workflows	2026	Wraps tool calls in progress-aware transactions: the runtime buffers effects, seals a transaction once its read/effect footprint is complete, and settles only after per-resource frontiers prove no earlier conflicting work can arrive, compensating reversible effects on abort and gating irreversible ones before release. Exposes the tension between isolation and the side effects real tools cannot take back, at microsecond-scale wrapper overhead.
Sandboxes fast enough to disappear
SpecBox — Speculative Sandbox Scheduling for Efficient LLM Agent Serving	2026	Speculative sandbox scheduling — start the sandbox before you are certain it will be needed.
DeltaBox — Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback	2026	Millisecond-level sandbox checkpoint and rollback for stateful agents.
Firecracker — Lightweight Virtualization for Serverless Applications	2020	The microVM that most agent sandboxes are built on, and the case for why a VM boundary can still be cheap.
Sandlock — Confining AI Agent Code with Unprivileged Linux Primitives	2026	Confining agent-generated code using unprivileged Linux primitives rather than a VM.
TClone — Low-Latency Forking of Live GUI Environments for Computer-Use Agents	2026	Makes workspace versioning a first-class primitive: sibling containers with copy-on-write memory sharing and filesystem versioning let a live GUI workspace be forked, rolled back, and selectively merged, so speculative or parallel agent branches share environment state. Fast branch creation is separated from durable checkpointing, cutting end-to-end agent-loop latency 1.9x over KVM and 1.5x over CRIU.
OS-level control over what an agent touches
AgentCgroup — Understanding and Controlling OS Resources of AI Agents	2026	Characterizes OS-level resource use of sandboxed coding agents — OS execution is 55-60% of task latency, memory rather than CPU is the concurrency bottleneck, and spikes are tool-call-driven and unpredictable — then argues container-level, user-space controls are the wrong granularity and too slow to react. The proposed fix pushes enforcement in-kernel with eBPF and cgroup hierarchies aligned to tool-call boundaries; note the evaluation is preliminary.
ActPlane — Programmable OS-Level Policy Enforcement for Agent Harnesses	2026	Splits policy declaration from enforcement — the agent declares policy in an information-flow DSL, eBPF enforces it in the kernel with semantic feedback rather than opaque errors — so indirect execution paths that bypass the tool layer are still covered, at 1.9-8.4% overhead. The hard part it names is the semantic gap: policy intent arrives as underspecified natural language while enforcement must act on concrete system actions.
agent libOS — A Runtime Substrate for Capability-Controlled Self-Evolving LLM Agents	2026	A library-OS framing where agent authority is explicit — typed capabilities, task-authority ceilings, and budgets gate operations, while a separate information-flow plane propagates labels and requires one-shot human release for sensitive egress — so a self-evolving action surface does not implicitly widen resource authority. Read it for the capability-security argument rather than the evaluation, which is small and source-bound, and note it explicitly disclaims preventing prompt injection or providing kernel-grade sandboxing.

Efficient agent serving systems II, continued: session state and agent memory

Folded into the Nov 23 meeting above. The schedule merged this lecture into agent serving II, and the cache half of the list below is assigned with the KV-cache discussion rather than here. Agents are stateful and long-lived, and their context comes back nearly unchanged on the next turn. The KV cache stops being a per-request scratchpad and becomes shared, persistent state with a lifetime policy — which makes it a storage problem more than an attention problem. Read Continuum for the session abstraction, then the supplementary entries below for where that state physically goes.

Paper	Year	Why read it
Sessions that outlive a request
Continuum — Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live also Nov 18	2025	Multi-turn agent scheduling with an explicit time-to-live on cached state: how long is a session’s KV cache worth keeping?
Talaria — Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs	2026	Takes the session-affinity versus elasticity conflict head on: serverless wants stateless workers, agent sessions want sticky KV. Shows what reconciling them costs at scale.
The KV cache as persistent state
Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving also Oct 7, Oct 21, Nov 4	2024	A KVCache-centric architecture in production at scale: disaggregation plus a distributed cache pool plus overload handling.
CacheGen — KV Cache Compression and Streaming for Fast Large Language Model Serving also Oct 19	2023	If cached KV has to cross a network, its encoding is a bandwidth problem, not just a memory problem.
prediction-based KV management — Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management also Oct 21	2026	Predict what a dynamic agent workflow will need next and manage the cache accordingly.
KV cache management survey — From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving also Oct 19	2026	From a tensor buffer to a distributed memory hierarchy — the survey to read before picking a presentation paper in this area.
Leyline — KV Cache Directives for Agentic Inference	2026	Adds a serving-side primitive for editing the KV cache mid-session: a declarative directive says which span to remove or replace and whether to splice in place or re-prefill from a trimmed prefix, with a closed-form RoPE-rotation correction restoring attention math so the suffix need not be recomputed. It targets agent sessions that retry failed tool calls and drop stale outputs, where harnesses otherwise pay full prefix recomputation on every edit.
CommitKV — Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents	2026	Uses tool-call commit boundaries as measurement points: it compares a KV page’s deletion effect before a commit and after the returned observation is absorbed, separating pages that are merely dormant from those whose role is genuinely finished — a distinction single-snapshot attention-score eviction cannot make.
Agent memory
MemGPT — Towards LLMs as Operating Systems also Sep 14	2023	Treats the context window as a memory hierarchy with explicit paging. The most systems-flavored agent paper on the list.
memory serving cost — Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems	2026	Benchmarks three agentic memory systems (Mem0, Hindsight, Mastra) against a rolling window and full-transcript resubmission over conversations up to 400 turns, pairing every cost measurement with LoCoMo accuracy. The systems lesson is that serving cost is not predictable from conversation length and message size — a regression fit to the reference strategies misses the memory systems by 18-69% — and break-even against simply resending the transcript ranges from tens of turns to never.
agent memory backends — MemoryLake on MemoryArena: A Matched Study of Agent Memory Backends	2026	Runs four agent-memory backends through one harness — same framework, model alias, task samples, and scoring code — on a benchmark that scores interdependent multi-session task completion rather than post-hoc recall. Read it as much for its own caveats: because each backend bundles retrieval, consolidation, and prompt-assembly choices, even a matched comparison cannot attribute the gap to any single design decision.

Efficient agent serving systems II, continued: multi-agent workloads and performance optimization

Folded into the Nov 23 meeting above. Many agents running over nearly identical context, and the measurement problem that comes with them. Every number in Part II is conditional on a workload, so the trace and simulator entries are here rather than in a corner: they are what Assignment 5 and the project evaluation are built on. Read TokenCake for the sharing opportunity and ScaleSim for how to sweep a design you cannot afford to build.

Paper	Year	Why read it
Sharing state across agents
TokenCake — A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications	2025	A KV-cache-centric serving framework built for multi-agent applications.
TokenDance — Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing	2026	Collective KV cache sharing across agents — scaling by exploiting how much they have in common.
ForkKV — Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache	2026	Copy-on-write over a disaggregated KV cache for multi-LoRA agent serving. The OS analogy is exact and deliberate.
Kairos — Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloud	2025	Many agents contending for one shared model under overload, where per-request scheduling is blind to the workflow: Kairos orders requests by each agent’s latency characteristics and dispatches by memory demand, so one queued call does not gate an entire workflow.
Many agents at once
ScaleSim — Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management	2026	Large-scale multi-agent simulation, scheduled by invocation distance.
large language monkeys — Scaling Inference Compute with Repeated Sampling	2024	Scaling inference compute by repeated sampling: coverage keeps rising with the number of samples. The cleanest statement of “spend more decode tokens, get more correct answers”, and therefore of why throughput is a quality knob.
CoAgent — Concurrency Control for Multi-Agent Systems	2026	Imports serializability into multi-agent execution but inverts the classic remedy: rather than blocking (2PL) or aborting (OCC), it fixes a serialization order up front, applies writes speculatively in place, and notifies the affected agent so the LLM itself re-judges and patches its plan, with pre-registered saga-style inverses undoing misordered writes.
Maestro — Workload-Aware Cross-Cluster Scheduling for LLM-Based Multi-Agent Systems	2026	Predicts each agent stage’s output length and memory footprint, then uses those predictions to drive a three-level scheduler: multi-model co-location via weight caching at the node, latency-aware routing across clusters to dodge cold starts, and workflow-aware prioritization globally to limit head-of-line blocking on interactive tasks.
Aries — Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework	2026	An experimentation framework for agentic serving, which matters because reproducing an agent-workload result currently means reproducing the whole harness and tool environment. Methodology reading alongside Etalon and Vidur.
Simulators
Vidur — A Large-Scale Simulation Framework For LLM Inference	2024	Simulate configurations you cannot afford to benchmark. The right tool for a large sweep on a small allocation.
Frontier — Towards Comprehensive and Accurate LLM Inference Simulation	2026	A discrete-event simulator for modern disaggregated serving (PD and attention-FFN) with stateful reasoning, agent, and RL-rollout workloads — built on Vidur’s DES core but refactored out of its monolithic-replica abstraction, for exploring a design space without burning GPU hours. Read the error analysis: the argument is that average-case analytical proxies can reverse optimization conclusions, not merely blur them.
when does disaggregation pay — When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference	2026	Simulates prefill-decode and four-way prefill-decode-attention-FFN specialization and finds disaggregation pays more on agentic traffic than on chat (1.82x vs 1.29x over unified execution), driven by the context that accumulates across turns. A clean example of workload shape driving hardware specialization.
Traces and benchmarks
BurstGPT — A Real-world Workload Dataset to Optimize LLM Serving Systems	2024	A real request trace, including the burstiness that synthetic Poisson arrivals leave out.
Mooncake trace — Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving also Oct 7, Oct 21, Nov 4	2024	The Mooncake paper ships a production trace; useful for cache and scheduling experiments.
Splitwise trace — Splitwise: Efficient generative LLM inference using phase splitting also Oct 7	2023	Production inference traces from Azure, split by phase.
Etalon — Holistic Performance Evaluation Framework for LLM Inference Systems	2024	Argues that mean TTFT and TPOT hide what users experience, and proposes fluidity-based metrics instead. Read it before choosing your project’s metric.
MLPerf — Benchmarks (MLCommons)	—	How the industry defines an inference benchmark, including the rules about what may be tuned.
KernelBench — Can LLMs Write Efficient GPU Kernels? also in the agents and LLMs-for-systems tracks	2025	Can a model write a fast GPU kernel? Directly relevant to Assignment 4.

Nov 30 Student sharing — what you optimized and what it cost (Assignment 5), 1 of 2

Nothing to read for this meeting. Bring your Assignment 5 numbers — one profile, one prediction, one measurement, and the gap between the last two. Assignment 5 gets two sharing meetings rather than one; this is the first.

Dec 2 Student sharing — what you optimized and what it cost (Assignment 5), 2 of 2; course wrap-up

Nothing to read for this meeting. The second half of the room presents the same four things, and then we ask what the semester’s numbers add up to. Last meeting of the semester.

Not Tied to a Class Meeting

Material no single meeting owns: the three self-study optional content tracks — working with agents, LLMs for systems, and the training side of the stack — the caching and eviction background behind the Nov 4 meeting and a good many final project questions, and the surveys, tools, and paper-reading advice worth having open all semester.

Self-Study: Working with Agents in Practice

Not lectured on — this is the working-with-agents optional content track, kept here because Assignments 1 and 2 are written against exactly this material: how you tell an agent what you want, how you check cheaply that it did it, and when a second agent buys anything beyond a second bill. Read SWE-agent for the interface argument — what the tools look like bounds what the agent can do — and the rest for self-correction, multi-agent conversation, and a benchmark whose task is close to what you will be doing yourself by Assignment 4.

Lecture notes for this track →

Paper	Year	Why read it
SWE-agent — Agent-Computer Interfaces Enable Automated Software Engineering also Sep 23; in the LLMs-for-systems track	2024	The agent-computer interface is the contribution: what the tools look like matters more than which model is behind them.
Reflexion — Language Agents with Verbal Reinforcement Learning also Sep 14, Sep 23	2023	Self-critique across attempts: the cheapest verifier is often the agent itself, and it makes trajectories long and stateful.
AutoGen — Enabling Next-Gen LLM Applications via Multi-Agent Conversation also Sep 23	2023	Multi-agent conversation as a programming model — and the scheduling problem it hands the serving system.
KernelBench — Can LLMs Write Efficient GPU Kernels? also Nov 23; in the LLMs-for-systems track	2025	What a specification looks like when the task is a kernel and the verifier is a compiler and a stopwatch. Directly relevant to Assignment 4.

Self-Study: LLMs for Systems Research I

Not lectured on — this is the LLMs-for-systems optional content track, and the premise behind Assignment 5: models that write, optimize, and evaluate systems code. Read these as systems papers — ask what the harness is, what the environment lets the agent touch, and what the benchmark actually scores. SWE-agent is the agent-computer interface argument; KernelBench is the one whose task is closest to what you did in Assignment 3.

Paper	Year	Why read it
Agents that write and fix code
SWE-agent — Agent-Computer Interfaces Enable Automated Software Engineering also Sep 23; in the agents track	2024	The agent-computer interface is the contribution: what the tools look like matters more than which model is behind them.
OpenHands — An Open Platform for AI Software Developers as Generalist Agents	2024	An open platform for coding agents — the practical reference implementation.
Codex — Evaluating Large Language Models Trained on Code	2021	Where code models and pass@k came from.
AlphaCode — Competition-Level Code Generation with AlphaCode	2022	Massive sampling plus filtering. A useful reminder of how much compute early results cost.
Agents that optimize systems code
KernelBench — Can LLMs Write Efficient GPU Kernels? also Nov 23; in the agents track	2025	Can a model write a fast GPU kernel? Directly relevant to Assignment 4.
KernelBench-Verified — Do LLM-Generated Kernels Actually Beat PyTorch?	2026	Extends KernelBench with a TF32-enabled baseline and a four-distribution hidden test suite, after finding that frontier models hardcode bypasses for the narrow test distribution. Under the tightened protocol the best model falls from a 1.43x to a 0.88x geomean speedup, and no model consistently beats PyTorch — essential context before trusting any kernel-agent number.
Astra — A Multi-Agent System for GPU Kernel Performance Optimization	2025	Optimizes existing CUDA kernels extracted from SGLang rather than translating PyTorch modules, with specialized agents iterating over code generation, testing, profiling, and planning for a 1.32x average speedup using zero-shot o4-mini. Note the harder starting point: the baseline is already-tuned production kernels, not stock PyTorch.
compiler optimization — Large Language Models for Compiler Optimization	2023	An LLM predicting optimization passes, evaluated against a real compiler.
AI-PROPELLER — Warehouse-Scale Interprocedural Code Layout Optimization with AlphaEvolve	2026	Evolutionary LLM search applied to production code layout at fleet scale with measured wins. The clearest evidence that LLM-driven optimization pays on a real, well-studied compiler problem.
Glia — A Human-Inspired AI for Automated Systems Design and Optimization	2025	Applies an agent to designing systems policies such as scheduling and load balancing rather than only writing code, and reports designs that remain interpretable. The most ambitious datapoint here and a strong discussion anchor.
VibeServe — Can AI Agents Build Bespoke LLM Serving Systems?	2026	A multi-agent loop that synthesizes entire LLM serving stacks end-to-end: an outer loop searches system designs, an inner loop implements candidates, checks correctness, and measures them on the target benchmark. It stays competitive with vLLM in the standard setting and beats generic stacks in six non-standard scenarios, arguing for generation-time specialization over runtime generality.
Benchmarks beyond the pull request
SWE-bench Multimodal — Do AI Systems Generalize to Visual Software Domains?	2024	Does any of it generalize outside Python?
BountyBench — Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems	2025	Agents on real security tasks, scored in dollars.
filesystem design benchmark — Benchmarking LLMs on File System Design and Implementation	2026	phi-Bench: 505 file-system tasks spanning understanding, implementation, performance modeling, debugging, optimization, and new-feature development, built from expert-written, textbook-adapted, and AI-generated tasks. A genuinely systems-flavored benchmark — read it for the reported causes of failed fs tasks across six open and proprietary models.
Learned components, and how to measure them
learned index structures — The Case for Learned Index Structures	2017	The ancestor of “replace a systems component with a model”, and still the clearest statement of the tradeoff.
measure one level deeper — Always Measure One Level Deeper	2018	Ousterhout on why end-to-end numbers alone will mislead you, and what to measure instead. Read it before Assignment 5.

Self-Study: LLMs for Systems Research II

Not lectured on — the second half of the LLMs-for-systems optional content track, and the same question one level up: can the loop run a research process rather than a coding task. Read the benchmarks for what they cannot see — a score that rises while the underlying claim goes unchecked is the failure mode both of these papers are trying to avoid, with mixed success. Useful directly for Assignment 5 and for judging your own project’s evaluation.

Lecture notes for this track →

Paper	Year	Why read it
Can the loop run a research process?
MLGym — A New Framework and Benchmark for Advancing AI Research Agents	2025	A gym-style environment for AI research agents, with tasks and an evaluation protocol.
AI Scientist — The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery	2024	The first version, and the better target for criticism of the framing.
AI Scientist-v2 — The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search	2025	Agentic tree search over the whole research loop. Read it skeptically and decide what you believe.
statistical model discovery — Automated Statistical Model Discovery with Language Models	2024	LLMs proposing and revising statistical models.
verification gap — Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap	2026	Surveys AI-scientist systems around the argument that generation has outpaced verification. A structured way to interrogate the AI Scientist-v2 and MLGym claims above.
Benchmarks, and what they cannot see
MLE-bench — Evaluating Machine Learning Agents on Machine Learning Engineering	2024	Agents on Kaggle competitions, with human baselines to compare against.
SWE-bench — Can Language Models Resolve Real-World GitHub Issues?	2023	The benchmark that reframed code generation as repository-scale issue resolution.
Multi-SWE-bench — A Multilingual Benchmark for Issue Resolving	2025	Extends SWE-bench to Java, TypeScript, JavaScript, Go, Rust, C, and C++ across 1,632 expert-annotated instances, and finds that agents strong on Python generalize poorly to the rest. The paper attributes the gap to a harder instance mix and Python-tuned scaffolds as much as to the languages themselves, which is itself the useful lesson about benchmark transfer.
SWE-Lancer — Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?	2025	Prices tasks in real dollars and grades with end-to-end tests, giving an economic denominator for agent capability that pairs with the cost-per-task framing used elsewhere in this course.
SWE-Bench ProMax — Benchmarking Agents on Large-Scale Multilingual Code Refactoring	2026	Moves from localized fixes to behavior-preserving refactors averaging 11.4 modified files and 261.6 lines across seven languages, where the best frontier model resolves only 41.2%. The long-horizon, wide-context regime it measures is exactly what stresses the serving system rather than just the model.
PaperBench — Evaluating AI’s Ability to Replicate AI Research	2025	Scores agents on replicating 20 ICML 2024 Spotlight and Oral papers from scratch against hierarchical rubrics co-developed with each paper’s authors, graded by an LLM judge — a harder and more research-realistic target than Kaggle-style tasks. The best tested agent reaches 21.0% and still trails the human PhD baseline; the rubric-and-judge methodology is itself worth critiquing.
OSWorld — Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments	2024	Agents driving a real desktop.
compression and generalization — What Fits (Into Few Tokens) Doesn’t Overfit: Compression and Generalization in ML Research Agents	2026	Shows that an ML research agent’s discoveries survive extreme compression: a fresh reproducer agent recovers the performance from only a very short prompt, and one-bit feedback is enough for the explorer to find good models across 8 datasets. The bottlenecks cost almost nothing, which supports a description-length explanation for why benchmark reuse overfits so little — and, when overfitting is deliberately induced, reproduction under short prompts breaks.

Self-Study: Distributed Training and Fine-Tuning

Not lectured on — this is the optional content track, kept here because projects that touch fine-tuning or distributed training need it. The Ultra-Scale Playbook is the best single entry point; these go deeper.

Paper	Year	Why read it
What a frontier training run looks like
Llama 3 — The Llama 3 Herd of Models self-study	2024	§3 is the one to read: 16K GPUs, real failure rates, and what actually breaks at scale.
DeepSeek-V3 — Technical Report self-study	2024	Mixture-of-experts at frontier scale with an unusually candid systems section (FP8 training, custom communication kernels, hardware co-design).
MegaScale — Scaling Large Language Model Training to More Than 10,000 GPUs self-study	2024	10,000+ GPUs, with the diagnosis and fault-tolerance machinery that scale requires.
Sharding weights and activations
Megatron-LM — Training Multi-Billion Parameter Language Models Using Model Parallelism self-study	2019	Tensor parallelism: split individual matmuls across devices. Start here.
ZeRO — Memory Optimizations Toward Training Trillion Parameter Models self-study	2019	Shard optimizer state, gradients, and parameters instead of replicating them. Start here.
FSDP — PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel self-study	2023	The PyTorch-native version, and an honest account of what was hard to make work.
Megatron on GPU clusters — Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM	2021	How tensor, pipeline, and data parallelism compose — the 3D-parallelism accounting.
Ulysses — DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models	2023	Sequence parallelism for very long contexts.
Alpa — Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning self-study	2022	Search the parallelization strategy instead of hand-picking it.
TorchTitan — One-stop PyTorch native solution for production ready LLM pre-training self-study	2024	A current, readable reference implementation of all of the above composed together.
Pipeline parallelism
GPipe — Efficient Training of Giant Neural Networks using Pipeline Parallelism self-study	2018	Pipeline parallelism with micro-batches, and the bubble that comes with it.
PipeDream — Fast and Efficient Pipeline Parallel DNN Training self-study	2018	Asynchronous pipelining with weight stashing — fills the bubble at the cost of staleness.
zero bubble — Pipeline Parallelism	2023	Splitting the backward pass to remove the pipeline bubble almost entirely.
Fitting it in memory
sublinear memory — Training Deep Nets with Sublinear Memory Cost self-study	2016	The original gradient-checkpointing result. Short and worth reading.
activation recomputation — Reducing Activation Recomputation in Large Transformer Models	2022	Selective recomputation: the memory/compute knob, tuned properly.
ZeRO-Infinity — Breaking the GPU Memory Wall for Extreme Scale Deep Learning self-study	2021	Offload to CPU and NVMe when the model does not fit in aggregate GPU memory.
Low precision and communication
FP8-LM — Training FP8 Large Language Models self-study	2023	FP8 training end to end, and which tensors cannot tolerate it.
Horovod — fast and easy distributed deep learning in TensorFlow	2018	Ring all-reduce as a usable library. Where data-parallel scaling became routine.
Fine-tuning
LoRA — Low-Rank Adaptation of Large Language Models self-study	2021	Low-rank adapters: fine-tune a small delta, keep the base frozen. Also what makes multi-tenant adapter serving possible.
QLoRA — Efficient Finetuning of Quantized LLMs self-study	2023	LoRA on a 4-bit frozen base — fine-tuning a large model on one GPU.

Self-Study: Post-Training and RLHF Systems

Also part of the self-study material. Included because the RL loop is a genuinely interesting systems workload: it runs generation and training in the same job, with the sampler feeding the trainer.

Paper	Year	Why read it
InstructGPT — Training language models to follow instructions with human feedback also Sep 2	2022	The RLHF pipeline, stated plainly.
PPO — Proximal Policy Optimization Algorithms	2017	The optimizer underneath most RLHF. Read the algorithm box, skip the rest if you like.
DPO — Direct Preference Optimization: Your Language Model is Secretly a Reward Model	2023	Removes the reward model and the RL loop entirely — a large systems simplification.
HybridFlow / verl — HybridFlow: A Flexible and Efficient RLHF Framework	2024	The systems paper of the group: how to place and schedule generation and training together.
OpenRLHF — An Easy-to-use, Scalable and High-performance RLHF Framework	2024	A practical open implementation built on Ray and vLLM.

Self-Study: Data Pipelines and Checkpointing

Part of the training-side self-study material. Checkpointing in particular is a storage-systems problem that gets solved badly surprisingly often.

Paper	Year	Why read it
Datasets and preprocessing
Data-Juicer — A One-Stop Data Processing System for Large Language Models self-study	2023	Data processing treated as a system with operators and a pipeline, not a pile of scripts.
Dolma — an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research self-study	2024	An open pretraining corpus with the curation decisions documented.
DataComp-LM — In search of the next generation of training sets for language models self-study	2024	Hold the training recipe fixed and vary only the data — a controlled experiment on data quality.
Checkpointing and storage
CheckFreq — Frequent, Fine-Grained DNN Checkpointing self-study	2021	Pick the checkpoint interval from first principles, and overlap the write with compute.
GEMINI — Fast Failure Recovery in Distributed Training with In-Memory Checkpoints self-study	2023	Checkpoint into the memory of other machines. Recovery time drops by orders of magnitude.
ByteCheckpoint — A Unified Checkpointing System for Large Foundation Model Development self-study	2024	A unified checkpointing system, including resharding when the parallelism configuration changes.
3FS — Design Notes (DeepSeek) self-study	2025	Design notes for the parallel file system behind DeepSeek’s training and inference pipelines.

Background: Caching and Eviction

Background for the Nov 4 prefix-cache meeting, and for any final project that touches eviction. A prefix cache is a cache: the objects have wildly different sizes and costs, the workload is skewed and non-stationary, and the eviction policy decides most of the hit ratio. These are the papers on how that problem is normally attacked, and what the standard mistakes are.

Paper	Year	Why read it
Eviction policies
S3-FIFO — FIFO Queues are All You Need for Cache Eviction	2023	Three FIFO queues beat LRU on hit ratio and on throughput. The clearest demonstration that “one-hit wonders” dominate real cache workloads.
SIEVE — is Simpler than LRU: an Efficient Turn-Key Eviction Algorithm for Web Caches	2024	Lazy promotion in about ten lines of diff over FIFO. A good model for how simple a competitive policy can be.
LRB — Learning Relaxed Belady for Content Distribution Network Caching	2020	Learn to approximate the offline optimum. Read it for the framing of eviction as prediction, and for how much machinery that costs.
TinyLFU — A Highly Efficient Cache Admission Policy	2015	Admission, not eviction, is often the lever — and a sketch is enough to make the decision.
Cache engines and simulators
libCacheSim — a high-performance cache simulator and library	—	The standard cache simulator for this kind of work. Read the docs before writing a policy.
Segcache — A Memory-Efficient and Scalable In-Memory Key-Value Cache for Small Objects	2021	Metadata overhead is the hidden cost when cached objects are small — which is exactly the regime of a per-block prefix cache.
CacheLib — The CacheLib Caching Engine: Design and Experiences at Scale	2020	What a production cache has to handle beyond the eviction policy: sizing, admission, flash, and warm restarts.
The same problem in LLM serving
Marconi — Prefix Caching for the Era of Hybrid LLMs also Nov 4	2024	Prefix-cache eviction when entries are not uniform — the closest published treatment of the problem the Nov 4 meeting sets up.
Mooncake — A KVCache-centric Disaggregated Architecture for LLM Serving also Oct 7, Oct 21, Nov 4, Nov 23	2024	The production view: a distributed KV cache pool with real capacity limits and real hit-ratio consequences.

Surveys and Long-Form References

When a topic is new to you, start with one of these rather than with a random recent paper. They are also the fastest way to find a presentation paper you actually want to read.

Paper	Year	Why read it
Ultra-Scale Playbook — The Ultra-Scale Playbook: Training LLMs on GPU Clusters	2025	The best single entry point to distributed training, with runnable intuition instead of only equations.
efficient inference survey — A Survey on Efficient Inference for Large Language Models	2024	A map of the whole Part II landscape: model-level, system-level, and hardware-level.
efficient LLMs survey — Efficient Large Language Models: A Survey	2023	Broader still, covering training as well as inference.
full-stack inference survey — Full Stack Optimization of Transformer Inference: a Survey	2023	Works down from the model to the hardware; good for seeing where the layers meet.
spec decoding survey — Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding also Nov 11	2024	Taxonomy of drafting and verification strategies.
OSTEP — Operating Systems: Three Easy Pieces	2023	Free, and the right refresher if virtual memory, paging, or scheduling feel distant. Several papers in this course are OS ideas applied to a GPU.

Documentation, Tools, and Engineering References

Not readings — things to have open while working on the assignments and the project.

Paper	Year	Why read it
vLLM docs — vLLM documentation also Oct 5	—	The serving stack most projects will build on.
SGLang — (source repository)	—	The other stack, and the reference implementation of RadixAttention.
Nsight Systems — NVIDIA Nsight Systems	—	Timeline profiling. The tool that shows you the gap between kernels you did not know was there.
PyTorch profiler	—	Lower-friction profiling from inside Python; enough for most Assignment 5 measurements.
libCacheSim — (source repository)	—	Cache simulator, and the quickest way to try an eviction idea.
ACM sigconf template — ACM Primary Article Template	—	The format to use for the final project report.

Reading and Evaluating Systems Papers

Presenting a paper is worth 12% of the grade. The paper discussion page has the presenter guide, the rubric, and a three-pass reading strategy; these are the two short pieces worth reading before your first paper.

Paper	Year	Why read it
How to Read a Paper — (Keshav)	2007	The three-pass method. Four pages, and it will save you many hours over the semester.
Always Measure One Level Deeper also in the LLMs-for-systems track	2018	Ousterhout on performance measurement: the mistakes are predictable, and this lists them.
Orca — A Distributed Serving System for Transformer-Based Generative Models also Oct 5	2022	Read again as a model of how to argue a systems result: one idea, cleanly isolated, honestly evaluated.

How to Use This Page

Reading by Class

Part I — Introduction to LLMs and Agents

Sep 2 Course overview and logistics

Sep 9 Modern ML basics: transformers

Sep 14 Agents from a user’s perspective

Sep 16 Agents from a designer’s perspective I

Sep 21 Student sharing — what you learned from using an agent (Assignment 1)

Sep 23 Agents from a designer’s perspective II

Further reading: the workload your design emits

Part II — Systems for LLMs

Sep 28 GPU programming for MLSys: architecture, memory, and kernels

GPU programming, continued: writing kernels — Triton, tiling, and fusion

Sep 30 Efficient LLM computing: GPU kernels

Oct 5 Efficient LLM serving: paging, batching, and scheduling I

LLM serving basics, folded: paging and the KV allocator

Oct 7 Efficient LLM serving: batching and scheduling II

Oct 14 Efficient LLM serving: routing and load balancing

Oct 19 Efficient LLM serving: KV-cache optimization I

Oct 21 Efficient LLM serving: KV-cache optimization II

Oct 28 Student sharing — what you optimized and what it bought (Assignment 3)

Nov 4 Efficient LLM serving: prefix cache

Nov 9 Efficient LLM serving: pruning and quantization

Efficient LLM serving: pruning and quantization, continued: the post-training line

Nov 11 Efficient LLM serving: speculative decoding

Nov 18 Efficient agent serving systems I

Efficient agent serving systems I, continued: scheduling agentic programs

Nov 23 Efficient agent serving systems II

Efficient agent serving systems II, continued: session state and agent memory

Efficient agent serving systems II, continued: multi-agent workloads and performance optimization

Nov 30 Student sharing — what you optimized and what it cost (Assignment 5), 1 of 2

Dec 2 Student sharing — what you optimized and what it cost (Assignment 5), 2 of 2; course wrap-up

Not Tied to a Class Meeting

Self-Study: Working with Agents in Practice

Self-Study: LLMs for Systems Research I

Self-Study: LLMs for Systems Research II

Self-Study: Distributed Training and Fine-Tuning

Self-Study: Post-Training and RLHF Systems

Self-Study: Data Pipelines and Checkpointing

Background: Caching and Eviction

Surveys and Long-Form References

Documentation, Tools, and Engineering References

Reading and Evaluating Systems Papers