This class sets up the question the rest of the semester answers: for a modern AI workload, where do the time and the money actually go? We draw the stack we will spend fourteen weeks taking apart, then do the one calculation that explains why most of this course is about serving rather than training. Logistics come last, on purpose. By the end you should be able to estimate how many generated tokens it takes before inference has cost more than training ever did.
This is the first of 24 meetings. Part I (6 classes) introduces LLMs and agents from a systems point of view, so that the rest of the semester has something concrete to optimize. Part II (18 classes, including three guest lectures) is the bulk of the course and works down the stack from framework to kernel to cluster to serving path, and closes the semester. What LLMs can do for systems research is optional content rather than a lectured block. Four student-sharing events are interleaved through the semester, after Assignments 1, 3 and 5 (Assignment 5's spans two meetings); the sessions for Assignments 2 and 4 are skipped. Most of the numbers that motivate Part II descend from the estimate in §1.6. The next meeting derives the 6ND rule that we use here on faith, and then goes on to split the 2N per generated token into prefill and decode and price them both.
75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221). The modern ML basics material (Lecture 2's notes) has its own meeting on Sep 9, so the budget below runs at full length rather than compressed beside it.
| Time | Segment | Notes |
|---|---|---|
| 0–5 | Open cold with the question | Write "where do the time and the money actually go?" on the board and leave it up all semester. No self-introduction yet. |
| 5–13 | §1.1 What this is and is not | Say "this is not an ML course" out loud. Some students are in the wrong room and deserve to know in the first ten minutes. |
| 13–21 | §1.2 Why now | Three changes plus the dated arc. Keep it to three. |
| 21–28 | §1.3 The stack, top to bottom | Six layers on the left board, annotated with dates. Leave the drawing up all class. |
| 28–36 | §1.4 One request, end to end | Thread a single arrow down that drawing, naming the resource at each hop. No quantities. |
| — | §1.5 Reading an accelerator datasheet | Reading-only, not scheduled. Assign it with the before-Sep 14 list. |
| 36–53 | §1.6 The break-even calculation | The centerpiece. Derive T = 3D live; no finished slide. |
| 53–59 | §1.7 Energy and carbon | Two conversions on the board, not a lecture on sustainability. |
| 59–69 | §1.8 How the course runs | Grading table on screen, not recited. Competition gets 90 seconds of its own. |
| 69–75 | §1.9 Expectations and this week | End on "what is it divided by?", then the before-Sep 14 list. |
If running long: cut §1.7 to the kWh line, and compress §1.8 to the grading table plus "read the five pages". Never cut §1.6 — it is the only part of today that has to land.
By the end of this class you should be able to:
N parameters and D training tokens, compute where cumulative serving compute overtakes training compute, and explain why the answer is independent of N.This is a systems course. The workload happens to be large language models and the agents built on them, but the questions are ones you already know how to ask: what is the bottleneck resource, what is on the critical path, what is cached and what is recomputed, and what one unit of useful work costs. We care about time, memory capacity, bandwidth, money, and energy.
We do not ask whether a model is good. That disappoints a few people every year, so be blunt: we never ask whether a model is accurate, only what it costs to train and to serve. If you want architectures, objectives, or learning theory, this is the wrong course. It is also not a prompt-engineering course — you will write prompts in Assignment 1, but as a way to generate a workload you can measure.
By the end you should be able to read a serving paper and predict, before its evaluation section, which numbers must be true and which are suspicious; size a deployment's memory and bandwidth on an envelope; and instrument an agent so that every token and second is accounted for. You will not leave with an opinion about which model to use, because that is a question about quality.
The prerequisite is one of CS61, CS1610, or CS2620, but what matters is comfort with memory hierarchies, caches, and concurrency, plus Python and PyTorch. Machine-learning concepts get explained as they arrive; systems concepts get used as tools. A KV cache is a cache, context management is an eviction policy, prefill and decode are a two-phase pipeline, and an agent is dependent-task scheduling. That transfer is the design of the course.
Minutes: 8.
Board: Two columns, "we ask" / "we do not ask". Left: FLOPs, bytes moved, GB, ms, dollars, joules. Right: accuracy, MMLU, "is the model good". Leave it up through §1.2.
Ask the room: "How many of you came here to make a model better, as opposed to cheaper?" Let the hands go up, then say the course is entirely about the second thing.
Expect confusion: Students conflate "systems for ML" with "ML". Fixing sentence: "We treat the model as a fixed workload with known arithmetic, the way a database course treats a query."
If short on time: Keep paragraph three; prerequisites are on the site.
The arc has checkable dates. The shape of the workload was fixed in 2017, when Attention Is All You Need replaced recurrence with attention and made the transformer the computation every layer of the stack now serves. In 2020, Scaling Laws for Neural Language Models turned model size from a discovery into a line item — loss falls predictably with compute, so a bigger model became something you budget for rather than something you find. In 2022, Chinchilla corrected the recipe: the compute-optimal token budget is roughly D ≈ 20N, far more data per parameter than practice had assumed. Chat then industrialized inference, and agents multiplied the number of model calls each user action generates. That sequence is why this course exists in 2026 and would have had little to teach in 2019: the workload, the budget, and the request pattern all changed after the systems underneath them were designed.
Three things changed, and each is a systems problem rather than a modeling one.
Models outran the device. For most of deep learning's history the interesting model fit on one accelerator, and the software stack assumed it. At 2 bytes per parameter, a model with N above roughly 40 billion parameters no longer fits in the 80 GB of an A100 or an H100 — weights alone, before anything else. Past that point, sharding weights, splitting layers across devices, and moving activations over an interconnect are not optimizations; they are the only way to run at all (now optional content).
Inference overtook training as the dominant lifetime cost. A training run is a large one-time expense with a known end. Serving is a bill that arrives every day for as long as anyone uses the model, and §1.6 shows how fast it wins. That inverts the priorities the field inherited from the era when a model was trained once and evaluated on a fixed test set, and it is why most of Part II is the serving path.
Agents turned one request into many dependent calls. A chatbot request was one prefill plus some decode steps. An agent request is tens or hundreds of model calls, each carrying a context that mostly repeats the previous call's, interleaved with tool invocations that have their own latency and failure modes. The unit of work became a trajectory rather than a request, so what systems people know about dependent tasks — critical paths, queueing, stragglers, caching of repeated work — now applies to a stack not built for it (Sep 16, Sep 23; Nov 18, Nov 23).
Minutes: 8.
Board: A timeline with three ticks: "model fits on one GPU" → "model does not" → "one request becomes a hundred". Anchor the left end with the publication years the body establishes — 2017 attention, 2020 scaling laws, 2022 Chinchilla — and label each tick with the lecture dates that own it.
Ask the room: "A 70-billion-parameter model in bf16 — does it fit on an 80 GB H100?" Get 140 GB said out loud, then note this is only the weights and Lecture 2 adds the rest.
Expect confusion: Students assume training is where the compute is, because that is what gets reported. Do not resolve it here; say "hold that thought" and let §1.6 do the work.
Common wrong answer: "Agents are chatbots with tools, so serving them is the same problem." Response: the context is re-sent on nearly every step, so the workload is dominated by repeated prefix processing, which chat barely exercises.
Here is the map we keep returning to: one request descending through six layers, each of which independently decides how much work gets done.
At the application layer, an agent decides how many model calls a task takes and how much context each carries — the layer with the largest available wins and the least attention paid to it. A loop that re-sends 20,000 tokens of unchanged context on each of forty steps has manufactured 800,000 tokens of prompt processing out of nothing (Sep 16, Sep 23). The framework layer — PyTorch and the engines built on it — turns a call into kernel launches and decides which tensors are materialized and which are recomputed (the next meeting, Sep 28). The kernel layer decides how many times each value crosses the boundary between on-chip memory and HBM; for attention the arithmetic did not change but the number of round trips did, and that was worth a large constant factor (Sep 30, Oct 5).
The accelerator sets a ceiling no software above it can exceed: an A100 80GB SXM offers 2,039 GB/s of memory bandwidth against 312 TFLOP/s of dense BF16 tensor throughput, an H100 SXM 3,350 GB/s against roughly 989 TFLOP/s dense. Across that generation, arithmetic got about 3.2× faster while memory got about 1.6× faster, so whether a piece of work benefits from the newer part depends on which of the two limits it (Sep 30, Oct 5). The interconnect and cluster layer carries what one device cannot hold, then handles failure at scale; its training-side material is now optional content, and the multi-replica half returns on Oct 26. The serving layer sits across all of it, deciding which requests share a batch, whether a prefix is already resident, which replica gets the request, and what precision the weights are in (Oct 7 through Nov 23).
Keep the map, because performance questions are almost always asked at the wrong layer. "Our inference is slow" is not a statement about a model: it has a different answer and a different fix at each of six levels.
Minutes: 7.
Board: Six layers as a vertical stack, top to bottom, with the units each is measured in beside it: tokens per task, kernel launches, bytes moved per FLOP, TFLOP/s and GB/s, GB/s across the link, requests/s at a latency target. Annotate with dates. Leave this up — §1.4 threads a request down it and §1.6 and §1.8 point back at it.
Ask the room: "A team says their agent is slow. Name three measurements that tell you which layer to blame." Push for tokens per completed task, time in tool calls versus model calls, achieved versus peak bandwidth.
Expect confusion: Students expect bottom-up ordering, as in an architecture course. Say that Part I goes top-down because you cannot judge a serving optimization before seeing the workload that motivates it.
Note: All TFLOP/s figures spoken aloud today are dense. The dense-versus-sparse caveat is written up in §1.5, which is reading rather than lecture; if it comes up, one sentence and point there.
If short on time: The six layers and their dates must survive; defer the A100/H100 ratio to Sep 30.
The map is six nouns. Here is what actually happens, in order, when you press enter on a chat request — one trip down the stack, naming at each hop the resource that gets consumed and the week that owns it. No quantities in this part; those are the job of the next meeting.
A tokenizer turns your prompt into token IDs. This is CPU work, and it is the only hop in the entire trip with no accelerator in it. It costs microseconds, nobody optimizes it, and it nonetheless decides S — the prompt length in tokens — which is the unit every later hop is billed in, in memory, in arithmetic, and in money. Tokens are not words. Whitespace, punctuation, source code, and non-English text all convert at different rates, so two prompts that look equally long to you can differ substantially as far as the machine is concerned. Every estimate you make this semester starts from a number this hop produced, so look at it rather than assume it.
The front end authenticates the request and puts it in a queue. No computation happens here and latency accrues anyway. How long the request waits is a policy decision rather than a physical one: how many requests the server admits at once, whether a short request may jump a long one, and which replica gets it at all. Those are Oct 7 and Oct 14, with replica choice on Oct 26. This is the part of a user's latency that has nothing to do with the model, and on a loaded server it is routinely the largest part: a serving system that is "slow" is very often a serving system that is queueing, and no kernel optimization touches that.
The scheduler admits it to a batch. The resource being allocated here is not compute — it is memory. Admitting a request means reserving room for the working state it will hold for its entire life, and that reservation, not the arithmetic, is what caps how many requests can run at once. Systems readers have met this before: it is admission control against a capacity constraint, the same shape as a connection pool or any server that keeps per-client state. Getting the policy right is Oct 7 and Oct 14; getting more requests into the same capacity is Oct 28.
Prefill. The prompt phase, to use the synonym once. The model runs one forward pass over all S prompt tokens at once. Because they all pass through the weights together, the work is large matrix-matrix multiplication, and the resource consumed is arithmetic: this is the compute-heavy phase of a request's life. The same pass writes the KV cache — for every layer, the keys and values that later attention steps will need to look back at. It is a cache in the ordinary systems sense, state kept so that work already done is not redone, and it is the single most recurring object in this course. The next meeting prices this phase; Sep 30 and Oct 5 are about making the kernels that execute it move fewer bytes.
Decode. The generation phase. The model now emits one token per step per sequence, and each step must read essentially the entire set of weights out of HBM to produce it. The resource is memory bandwidth, not arithmetic — the same weights that prefill pushed thousands of tokens through are now read to produce one. Recall §1.2's 70-billion-parameter model, whose bf16 weights alone are 140 GB: every decode step re-reads more bytes than fit on one device. Meanwhile the KV cache grows every step, adding a key and a value in every layer, for as long as generation runs. A request therefore gets more expensive to hold the longer it runs, which is unlike most caches you have met, where the cost of an entry is fixed at insertion. Batching many sequences into a single pass over the weights is the lever that makes any of this affordable, and it is the reason the scheduler in the previous hop exists at all. The next meeting quantifies the phase, Oct 7 and Oct 14 schedule it, Oct 28 handles the growth, and Nov 9 changes how many bytes "the weights" even is.
Detokenize and stream back. Each token is turned back into text and pushed to the client as it is produced, which is why output appears a piece at a time rather than all at once, and why serving has two latency metrics rather than one. Time to first token (TTFT) covers tokenization, queueing, and prefill; it is dominated by the length of your prompt and by how busy the server was when you arrived. Time per output token (TPOT) — inter-token latency, ITL, is the synonym — is the gap between the tokens that follow, and it is decode's bill, one step at a time. Nearly every decision in Part II trades one against the other: a larger batch raises tokens per second per GPU and makes somebody wait longer. The next meeting puts numbers on both. Here only the split matters.
The request ends and gives its memory back. The KV cache is freed and the slot returns to the scheduler. Until then the request holds capacity whether or not it is doing anything, which is why an abandoned browser tab with an open stream is not free and why timeouts are a capacity mechanism rather than a courtesy.
Now run that whole trace forty times over, with a context that barely changes between calls, and you have an agent (Sep 14, Sep 16, Sep 23). Every hop repeats. S grows on each step instead of resetting; the scheduler sees a burst of related requests rather than one; and prefill re-processes a prefix it has already processed, again and again. That last observation — that most of the prompt tokens in an agent workload have been through the machine before — is the entire motivation for Nov 4. Assignment 2 makes you write that loop, and Assignment 3 makes you account for it.
Minutes: 8.
Board: Do not draw anything new. Thread a single arrow down the six-layer stack from §1.3, stopping at each hop to write the resource beside it: CPU, queue, memory reservation, arithmetic, bandwidth, network. Six labels and nothing else. If anyone asks how many milliseconds, the answer is "later today".
Ask the room: "Between pressing enter and seeing the first token, how much of that time is the model computing?" Most of the room will say all of it. Queueing and prefill share it, and the split depends on load — which is why TTFT is as much a scheduling metric as a model one.
Expect confusion: Students hear decode as "prefill, repeated". Fixing sentence: prefill puts thousands of tokens through the weights in one pass, decode puts one through per pass — same weights, opposite bottleneck.
Common wrong answer: "The KV cache is an optimization you could switch off." Without it every step recomputes the whole prefix from scratch, so no practical server has the option. It is a cache in the sense that it stores reusable work, not in the sense that it is discretionary.
If short on time: Prefill versus decode and the TTFT/TPOT split must survive. The tokenizer and the teardown can be one sentence each.
This section is reading rather than lecture, and it teaches a skill you will use in every assignment and every time a paper reports a number that looks wrong. Datasheets are written to be quoted selectively. Read one in this order.
Memory bandwidth first. For the serving-dominated workloads of this course it is the number most likely to bind, because decode streams the weights past the arithmetic units once per generated token, and that is where nearly all tokens come from. The A100 80GB SXM offers 2,039 GB/s and the H100 SXM 3,350 GB/s. Bandwidth is also the hardest number on the page to inflate: it falls out of bus width and clock rather than out of a benchmark protocol or a choice of numeric format, so there is no favorable row to select. The next meeting pairs it against the arithmetic figure to get the single number you will carry all semester, and Oct 5 applies that number to real kernels.
Treat the headline throughput as marketing. The figure on page one is the 2:4 structured-sparsity number, which is exactly twice the dense figure on both reference parts: 624 against 312 TFLOP/s on the A100, 1,979 against roughly 989 on the H100. Structured sparsity requires that two of every four weights in a group be zero, in that pattern, so that the hardware can skip them — which a trained dense transformer's weights emphatically are not. Getting there means pruning and retraining and accepting whatever happens to model quality, so it is a modeling decision with a hardware payoff rather than a feature you switch on. Read the dense row. Then check which precision that row is: on the H100, FP8 dense is roughly 1,979 TFLOP/s, double the BF16 figure, and the A100 does not support FP8 at all. A comparison that quotes H100 FP8 against A100 BF16 is a 6.3× ratio (1,979 / 312) built from three changes at once, and it will be presented as one.
Check the form factor. The same silicon ships as an SXM module and as a PCIe card, and in practice they are not the same part: SXM gets the higher power envelope, the full NVLink fabric, and the memory bandwidth this course's canonical numbers assume, while the PCIe card runs at a lower power limit with reduced bandwidth and reaches its host over PCIe. Every number in these notes is for the SXM parts. When a paper's measurements look inexplicably low, form factor is a standard first thing to check. The interconnect line deserves the same suspicion — the A100's 600 GB/s (3rd-generation NVLink) and the H100's 900 GB/s (4th-generation) are useless until you know what they are per: per GPU or per link, one direction or the sum of both. A factor of two hides in each of those questions, and it starts to matter the moment a model is sharded across devices (now optional content).
Read TDP for exactly what it is. The 400 W on the A100 and the 700 W on the H100 are board power limits — ceilings the parts will not exceed. They are not measured draws and they are not performance figures. TDP has two honest uses: multiply it by hours for a worst-case energy bound, exactly as §1.7 does with 0.7 kW, and use it to size power and cooling. What it cannot tell you is what your workload actually draws, which depends on which units are busy; a bandwidth-bound decode loop that leaves the tensor cores mostly idle does not draw what a compute-bound prefill draws. If you need the real figure, measure it, which is what the instrumentation section of Sep 23 is for.
TDP is also worth dividing into the other two headline numbers, because the result says something the datasheet does not intend to. Arithmetic per watt went from 312 / 400 = 0.78 TFLOP/s per watt on the A100 to 989 / 700 = 1.41 on the H100, a gain of about 1.8×. Bandwidth per watt went from 2,039 / 400 = 5.10 GB/s per watt to 3,350 / 700 = 4.79, a change of about −6%. One generation therefore made arithmetic dramatically cheaper in energy terms and moving bytes very slightly more expensive. If your workload is bandwidth-bound — and the decode hop of §1.4 is — the newer part buys you speed rather than efficiency. Hold on to that; it is one of the reasons Part II is as long as it is.
Finally, "80 GB" is the opening of a negotiation, not an allowance. Weights claim their share first, and §1.2's 140 GB for a 70-billion-parameter model in bf16 already exceeds one device, so that model has no single-GPU deployment at that precision — the fleet's unit of replication is at least two GPUs before any other consideration. Then subtract, in roughly this order: the driver context and the framework's own allocations, which are charged before your first tensor exists; activation workspace for whatever is currently executing, which grows with batch size and sequence length; the KV cache of every in-flight request, held for that request's entire life; and finally the blocks the allocator is holding but cannot hand out in the size the next request wants. That last one is external fragmentation in the OSTEP sense, and it is exactly why the serving systems of Sep 28 borrowed paging from operating systems. Plan against the usable figure rather than the printed one, and decide deliberately what fraction goes to KV cache, because that fraction sets the maximum batch size and the batch size sets your throughput. The next meeting does the sizing.
One closing warning that applies to every line above. Every number on a datasheet is a ceiling under conditions the vendor chose, and none of them is a measurement of your workload. The distance between the ceiling and what you actually achieve is roughly the subject of this course, and you cannot read it off a page — you have to measure one level deeper than the layer you are trying to explain.
Minutes: 0 — reading-only, not scheduled. Assign it with the before-Sep 14 list in §1.9.
Board: Nothing. If sparsity or SXM-versus-PCIe comes up during §1.3, give one sentence — "the headline number is the 2:4 sparse figure, and dense is what a transformer gets" — and point here.
Expect confusion: Students read 80 GB as a budget and size a deployment with it. In office hours, walk the subtraction in order: weights, then framework overhead, then activation workspace, then per-request KV cache, then fragmentation — and only then ask what batch size fits.
Common wrong answer: "The H100 is three times the A100, so it is three times better." It is 3.2× the arithmetic and 1.6× the bandwidth, and which one you get is a property of the workload rather than of the part. Exercise 2 makes them find that out with their own arithmetic.
If short on time: Nothing to cut — it is not on the clock. If the room is ahead of schedule, the per-watt pair is the 90 seconds worth spending.
Now the calculation that organizes the course. For one model over its deployed life, compare the compute spent training it against the compute spent serving it. Both sides reduce to a FLOP count. Training a dense transformer with N parameters on D tokens costs about 6ND FLOPs — roughly 2ND forward and 4ND backward, derived in Lecture 2. Generating one token in the decode phase costs about 2N FLOPs per sequence: one forward pass, each parameter contributing a multiply and an add.
T is the total number of tokens generated over the model's life.
Set the two equal: 2 · N · T = 6 · N · D. Divide both sides by 2N: T = 3 · D.
N cancels, so the break-even point does not depend on model size — both training and serving are linear in N. The factor of 3 has a plain reading: a training token pays for a forward pass and a backward pass, the backward being about twice the forward, so one training token costs about what three generated tokens cost.
Substitute the Chinchilla compute-optimal ratio D ≈ 20N (Chinchilla): T = 3 · 20N = 60 · N.
Sixty generated tokens per parameter. Cumulative serving compute passes training compute there, and every token after that only widens the gap.
Put numbers on it with N = 70 × 10⁹. Compute-optimal data is D = 20N = 1.4 × 10¹² tokens, so C_train = 6 × 70 × 10⁹ × 1.4 × 10¹² = 5.88 × 10²³ FLOPs, and break-even output is T = 3D = 4.2 × 10¹² tokens. Check from the other side: 2 × 70 × 10⁹ × 4.2 × 10¹² = 5.88 × 10²³ FLOPs, as required. At 300 output tokens per response, 4.2 trillion tokens is 14 billion responses; at 10 billion generated tokens per day it arrives in about 420 days. Substitute your own numbers — anything with real usage crosses this line, after which every further day of serving spends more than the training run ever did.
What the estimate ignores, and which way each omission pushes:
2N FLOPs, so counting it makes serving more expensive and break-even in generated tokens arrives sooner than 3D. The next meeting separates prefill from decode properly.6ND for one successful run omits, and that pushes the other way.This is an order-of-magnitude argument, not an accounting statement. It survives the corrections because it is not close, and because the two largest push toward serving. That is why roughly two-thirds of this course is about inference.
Minutes: 17. The centerpiece; protect the time.
Board: Write C_train ≈ 6ND above C_serve ≈ 2NT, cancel N in front of the room, box T = 3D. Only then substitute D ≈ 20N and box T = 60N. Do the 70 × 10⁹ instantiation as a separate block — students copy numbers they watched appear. No slide with the answer on it.
Ask the room: "Before I substitute anything — does the answer depend on the model being large?" Wait for someone to see that N cancels. That moment is the lecture.
Expect confusion: Many students read T = 3D as "serving costs three times training". It is the opposite: serving is cheaper per token, which is exactly why it takes 3× as many tokens to catch up — and why catching up is inevitable. Say that out loud.
Common wrong answer: "Training dominates, because that is the number in the news." Answer with rates: training is a one-time integral, serving is a rate, and a rate beats an integral given time.
If short on time: Trim the caveats to prefill and utilization. T = 3D and T = 60N must be on the board, and 5.88 × 10²³ must survive because §1.7 reuses it.
The same FLOP count converts into hours and into joules with one multiplication each. Take the 5.88 × 10²³ FLOPs from §1.6. At 989 TFLOP/s dense BF16 and a 40% utilization assumption — an assumption, and a generous one — the effective rate is 3.96 × 10¹⁴ FLOP/s, so the run takes 5.88 × 10²³ / 3.96 × 10¹⁴ ≈ 1.49 × 10⁹ seconds, or 1.49 × 10⁹ / 3600 ≈ 4.1 × 10⁵ accelerator-hours. At $2 per GPU-hour, substitute your own rate, that is order $0.8M. An H100 SXM has a 700 W board power limit, so the same hours are about 4.1 × 10⁵ × 0.7 = 2.9 × 10⁵ kWh — roughly 290 MWh, and only the accelerator boards: no host CPUs, memory, network, storage, or cooling.
Multiply by your facility's overhead and your grid's carbon intensity — both lookupable for a specific deployment, and neither invented here — and you have an operational footprint. This belongs in a systems course rather than a policy discussion because of the framing in Sustainable AI: the footprint has an operational term from energy consumed while running and an embodied term from manufacturing the hardware, and the embodied share is too large to treat as a rounding error. One consequence people miss is that buying more accelerators to finish a job faster is not free even when energy per unit of work falls.
The practical point is simpler. Every optimization in Part II reduces FLOPs executed, bytes moved, or devices occupied per unit of delivered work, and each reduction is simultaneously a cost, a latency, and an energy reduction. Efficiency work here is sustainability work, not because we attach a moral frame to it, but because it is the same quantity over the same denominator.
Minutes: 6.
Board: 4.1 × 10⁵ h × 0.7 kW ≈ 2.9 × 10⁵ kWh. Write "boards only" beneath it and underline twice.
Ask the room: "What is missing from 290 MWh?" Collect host, network, storage, cooling, and the energy to build the machines.
Expect confusion: Students want a carbon number. Refuse, and explain why: it is a product of grid intensity, facility overhead, and hardware lifetime, none of which are properties of the model. Naming the factors is the teachable content; a fabricated total is not.
If short on time: Keep the kWh line and the last paragraph.
Twenty-four meetings: 12 instructor lectures, five student-led paper discussions, three guest lectures, and four student-sharing events, plus a final-project poster and demo session in December. Lectures carry the fundamentals; the research-heavy meetings marked Students in the schedule run as paper discussions. There is no midterm and no exam — the five assignments carry 70% of the grade, and the last one is due at the semester's last meeting.
| Component | Weight |
|---|---|
| Assignment 1 — use an agent | 10% |
| Assignment 2 — design an agent, plus the contributed problem set | 14% |
| Assignment 3 — optimize the agent | 18% |
| Assignment 4 — serve your own agent | 10% |
| Assignment 5 — optimize the full stack | 18% |
| Final project | 12% |
| Paper presentation | 12% |
| Class participation | 6% |
| Bonus — course feedback 5%; problems current AI cannot solve, 2% each | up to 10% |
The five assignments are 70% of the grade — 10% each for Assignments 1 and 4, 14% for Assignment 2 (10% for the agent, 4% for its problem set), 18% each for the two optimization assignments, 3 and 5 — and they tell one story. Assignment 1 (due Sun Sep 20) has you use Claude Code to build an agent console — a web app on AWS that runs tasks through Claude Code headless and records what each run cost — run example tasks through it, and report where the agent failed; those tasks seed Assignment 2's task set. Assignment 2 (due Sun Oct 4) makes you write the loop yourself against a commercial API and pin down the five numbers everything after it is measured against — and, alongside the loop, four problems an agent can be asked to solve, one easy, two medium, one hard, each with a scorer, difficulty measured by five runs of your own agent. The class's submissions are pooled into the held-out set that Assignments 3, 4 and 5 are graded against. Assignment 3 (due Sun Oct 25) holds those numbers still except for cost and latency: you are still renting the model, so every lever is one you wrote. Assignment 4 (due Tues Nov 10) takes the frontier model away — you serve a 7–8B class open-weight model yourself and recover the quality you lost, in the harness. Assignment 5 (due Wed Dec 2) does the optimization again with the serving stack finally in scope: profile it, commit to a prediction in writing before you touch the code, then optimize and measure against the same task set. That sequence is the course in miniature: build it, make it cheap from outside, own it, then make it cheap from inside — and once, build the ruler. Deadlines are 11:59pm on the day the schedule names, and the next assignment goes out at the following meeting, so the handoffs are Sep 21, Oct 5, Oct 26, and Nov 11. The first three go out during Part I and early Part II; take the release dates and the specs from the assignments page, which is the authoritative source for both. All five are individual.
Three meetings are student sharing sessions — Sep 21 for Assignment 1, Oct 28 for Assignment 3, and Nov 30 continuing into the last meeting on Dec 2 for Assignment 5, which is therefore presented before it is submitted. You show the room what you built, what broke, and one number you did not expect. Assignments 2 and 4 do not get one: Assignment 2 is the design assignment whose material comes back in Assignment 3, and Assignment 4’s bring-up work is reported in its write-up. Nothing formal, nothing separately graded; it is where the 6% participation is earned alongside the discussions. Build with that session in mind.
The final project is worth 12% and is the one piece of work where you choose the question. It is announced Oct 26, a one-page proposal is due Oct 28, the report and repository are due Dec 8, and there is a poster and demo session in December. Half the project's 12% is scored by four randomly assigned classmates at that session — and you owe four reviews of your own, due the same evening, each carrying scores and written feedback. Your score is the median of your four reviewers, not the mean, every review is signed to the instructor, and a review you skip costs you a quarter of your own peer-evaluated credit. One shape: build a tool that does not already exist — for your research, your studies, your week, or as an extension of one of the assignments, pushed somewhere the assignment never went — solving a problem no existing system solves. The bar is checkable: try the three closest things that already exist and name what each one cannot do. The other half is evidence: measure your tool on your own workload against two baselines, the closest existing system and you doing the job by hand, and report the comparison that went against you next to the ones that did not. There are deliberately no worked examples on the project page; finding the gap is half the work. A negative result, honestly measured, scores as well as a positive one. Take the deliverables and the rubric from the project page and nowhere else.
Two subjects are also available as optional self-study: working with agents in practice, a complete note that Assignments 1 and 2 draw on, and the training side of the stack. Both are listed under optional content. The Sep 28 and Sep 30 GPU meetings are part of the taught course; their extra reading and exercises are depth, not a separate optional track.
Minutes: 10.
Board: Nothing. Grading table on screen. Do not read it aloud; point at it and name the two things students get wrong — that there is no exam, with the five assignments and the project carrying the weight instead, and that the project is graded on whether the numbers are trustworthy, not on whether they are impressive.
Ask the room: "Which assignment do you think is hardest?" Most say Assignment 5. It is Assignment 4 — bring-up is where the semester's schedule actually breaks. Say it now so it is not a surprise in week eight.
Expect confusion: Students ask how ambitious the project has to be. Correct the framing — a sharp small question measured well beats a broad one measured badly — and send them to the project page for the deliverables rather than quoting them from memory.
If short on time: Grading table, the five assignment deadlines, and "read the five pages this week".
Read before class. In the second half of the semester that is not a suggestion — a discussion class with an unprepared room does not work, and everyone can tell within four minutes. "Read" means something specific here, and the paper discussion page spells it out as a three-pass strategy adapted from Keshav. Reading fewer papers more carefully beats skimming many. You are not expected to understand every detail; you are expected to arrive having formed an opinion — one thing you did not believe, one thing you would have done differently.
Participation is 6% and it is earned there, and at the four sharing sessions. One sharp question about how a baseline was configured is worth more than five comments; bring it written down, because that makes it far more likely you will ask it. AI tools are permitted throughout, under two conditions: disclose what you used, and take responsibility for what it produced. That, late days, and attendance are on the policy page, which also hosts the FAQ.
The single most useful habit to build this semester: when you see a performance number, ask what it is divided by. Tokens per second, per what — per GPU, per dollar, per watt, at what batch size, at what sequence length, at what percentile? "3× faster" is not a claim until you know the denominator and the baseline. Most disagreements in this field, and most mistakes in write-ups, are a missing denominator.
Before Sep 14: read the assignments, project, discussion, and policy pages once through — they are short, and they are the contract. The project is not due for months, but skimming it now is how you notice a tool worth building. Skim the Transformer paper if you have not read it. Compute is expected to come from Harvard FASRC (Cannon GPU partitions) and CloudLab, both to be confirmed; access instructions go on the project page once they are, so do not plan around an allocation that has not been announced. The TA and office hours are TBD and will be announced on the home page.
Minutes: 6.
Board: Write "÷ what?" in large letters beside the opening question and leave both up to the end.
Ask the room: "I tell you my kernel is 3× faster. What do you ask me?" Take answers until someone says "faster than what" and someone says "per what". Those two questions are the method of the course.
Expect confusion: Students hear "all readings optional" as "no work this week". Reframe: the reading is optional, the five pages of course mechanics are not.
If short on time: The "divided by what" habit and the before-Sep 14 list.
6ND FLOPs; generating a token costs about 2N. Serving compute overtakes training compute at T = 3D generated tokens, or 60N under D ≈ 20N, independent of model size.| Quantity | Value | Source |
|---|---|---|
| A100 80GB SXM — memory bandwidth | 2,039 GB/s (≈2.0 TB/s) | NVIDIA datasheet |
| A100 80GB SXM — dense BF16 tensor throughput | 312 TFLOP/s | NVIDIA datasheet |
| H100 SXM — memory bandwidth | 3,350 GB/s (3.35 TB/s) | NVIDIA datasheet |
| H100 SXM — dense BF16 tensor throughput | ≈989 TFLOP/s | NVIDIA datasheet |
| Training compute | ≈ 6ND FLOPs | Derived in Lecture 2 |
| Decode compute | ≈ 2N FLOPs per generated token | §1.6 |
| Chinchilla compute-optimal token budget | D ≈ 20N | Chinchilla |
| Training/serving break-even | T = 3D ≈ 60N generated tokens | Derived in §1.6 |
D = 20N = 1.4 × 10¹¹, so T = 3D = 4.2 × 10¹¹ — about 420 billion generated tokens. Equivalently 60N = 60 × 7 × 10⁹ = 4.2 × 10¹¹.N?Both sides are linear in N — 6ND and 2NT — so N cancels. Doubling the model doubles the training cost and the per-token serving cost in the same proportion.5.88 × 10²³ FLOPs. What fraction of peak dense BF16 must you sustain?Peak is 1,000 × 989 × 10¹² = 9.89 × 10¹⁷ FLOP/s over 30 × 86,400 = 2.592 × 10⁶ s, so 2.56 × 10²⁴ FLOPs are available. Required: 5.88 × 10²³ / 2.56 × 10²⁴ ≈ 0.23, about 23% of peak.70 × 10⁹ × 2 = 140 GB of weights alone exceeds one device, and the driver context, the activation workspace, every in-flight request's KV cache, and allocator fragmentation are all still unpaid out of what remains. 80 GB is the opening of a subtraction, not a budget (§1.5).These want a calculator and ten minutes each. Show the intermediate quantities, not just the answer — that is the habit being trained.
D ≈ 20N to reach T = 60N. Real deployments frequently do the opposite, training a smaller model far past compute-optimal precisely so that serving it is cheap. Take a 7-billion-parameter model (N = 7 × 10⁹, as in Self-check 1) trained on D = 15 × 10¹² tokens. Where is break-even, in generated tokens and in tokens per parameter? By what factor is it further out than the Chinchilla answer, and why is T = 3D still exactly true when T = 60N is not?Solution sketch: T = 3D = 4.5 × 10¹³ generated tokens, which is 4.5 × 10¹³ / 7 × 10⁹ ≈ 6.4 × 10³ tokens per parameter against 60 — about 107× further out. That factor is just the token budget's own excess: D/N = 15 × 10¹² / 7 × 10⁹ ≈ 2,143 against 20, and 2,143 / 20 ≈ 107. T = 3D never assumed anything about D; only the 60N form did, by substituting one particular training recipe. The trade is deliberate — training is paid once, while the smaller N cuts the ≈2N cost of every token served forever.
5.88 × 10²³ FLOPs of the 70B training run into H100-hours at a 40% utilization assumption. Redo it for the A100 80GB SXM under the same assumption. How many A100-hours? Where does the ratio between the two figures come from, and why is it not the ratio of the memory bandwidths? Convert both to board-level energy and check the result against the per-watt figures in §1.5.Solution sketch: 312 TFLOP/s × 0.40 = 1.248 × 10¹⁴ FLOP/s, so 5.88 × 10²³ / 1.248 × 10¹⁴ ≈ 4.71 × 10⁹ s ≈ 1.31 × 10⁶ A100-hours. Against the H100's unrounded 4.13 × 10⁵ hours that is a factor of 3.17, which is exactly 989/312 — the dense arithmetic ratio, because this conversion never consults bandwidth at all. It assumes the run is arithmetic-limited and hides everything else inside the 40%. Energy: 1.31 × 10⁶ × 0.4 kW ≈ 5.2 × 10⁵ kWh against the H100's 2.9 × 10⁵ kWh, a factor of 1.81 — the arithmetic-per-watt ratio from §1.5, and necessarily so, since hours scale as 1/throughput and energy as hours × TDP. The load-bearing assumption is that 40% is the same number on both parts, and §1.5's ratios are a reason to doubt it.
2 × 10¹⁰ output tokens per day from a 70-billion-parameter model in bf16. Using decode's ≈2N FLOPs per generated token, how many H100s would that take if decode ran at 100% of dense peak? Name three effects from this lecture that make your answer a floor rather than an estimate, with directions. One of them is not about utilization at all — what does 140 GB of weights do to the shape of the fleet?Solution sketch: 2 × 70 × 10⁹ = 1.4 × 10¹¹ FLOPs per token; × 2 × 10¹⁰ = 2.8 × 10²¹ FLOPs per day; / 86,400 = 3.24 × 10¹⁶ FLOP/s; / 9.89 × 10¹⁴ = about 33 H100s. A floor, because prefill is uncounted (adds), decode achieves a small percentage of dense peak rather than 100% (multiplies, and the next meeting shows this is the dominant correction by an order of magnitude), and a fleet sized to the daily mean carries no headroom for peaks (adds). The shape point: 140 GB does not fit in 80 GB, so the unit of deployment is at least two GPUs — roughly 17 two-GPU replicas rather than 33 independent ones (§1.5). The token count is known to a few percent and the utilization only to within a factor of ten, so a fleet estimate is a claim about utilization wearing a claim about tokens as a disguise.
Solution sketch: Quote the H100's FP8 dense row against the A100's BF16 dense row — 1,979 / 312 ≈ 6.3× of headroom to hide in, and the A100 cannot run FP8 at all, so the workload changed; quote a 2:4 sparse row on one side only; measure with the latency target removed, or at a batch size the A100's 80 GB could not hold; or report per node against per GPU. Ask instead for tokens per second per GPU at a fixed latency percentile, same precision, same batching policy, baseline tuned by someone who wanted it to win. The bound this lecture gives you for free: at equal precision neither peak ratio exceeds 3.17, so any per-GPU claim above that comes from something other than the two parts — a better-configured contender, a worse-configured baseline, or a changed workload — and the report should say which.
Attention Is All You Need — optional. Read §3, and within it §3.2 (attention) and §3.3 (the position-wise feed-forward network); essentially all of the FLOPs and the memory traffic live in those two. Table 1, on per-layer complexity and path lengths, is the one table a systems reader should linger on. Skip §6 — the BLEU scores are not why we read this. Question to hold: which tensors in a layer have to be kept around, and for how long?
Scaling Laws for Neural Language Models — optional. Read the introduction and the summary of findings, study the loss-versus-compute plots, ignore the appendices. Its relevance today is that these curves made compute something you budget rather than something you discover. Question to hold: in each curve, what is held fixed? The compute-optimal ratio was later revised — the D ≈ 20N we use comes from Chinchilla, not from this paper.
Sustainable AI — optional. Read for the framing rather than the figures: the split between operational and embodied footprint, and the argument that a life-cycle view changes which optimizations look worthwhile. Question to hold: which of these costs already appears in a metric your serving system reports, and which is invisible to it?
If you read only one: the Transformer paper, if you have never read it, because Lecture 2 assumes you know the shape of the computation. Otherwise the scaling laws paper, for the framing behind §1.6.
The next meeting is the long part. It derives the 6ND rule from the forward and backward pass, does the training-memory accounting that shows why weights are the smallest part of a training footprint, then splits the 2N per token into prefill and decode, introduces the latency metrics and the roofline, and turns today's FLOP ratio into a serving cost model. Monday, Sep 14 puts agents in front of you as a user — what they are, what they cost you, and how you would tell whether one worked. Sep 16 and Sep 23 turn that around to the designer's side: first the loop, the tool interfaces, and context management as a cache policy, then the specification the agent is held to, the verifier inside the loop, and the task set that says whether either worked. Part II then descends the stack from §1.3: serving basics (Sep 28), GPU programming and kernels (Sep 30, Oct 5), batching and scheduling (Oct 7, Oct 14), routing and load balancing (Oct 26), KV-cache and prefix caching (Oct 28, Nov 4), pruning and quantization (Nov 9), speculative decoding (Nov 11), and then two classes on agent serving (Nov 18, Nov 23). Training-side topics — parallelism, ZeRO, failures at scale — and what LLMs can do for systems research are optional content rather than lectures. The semester closes with the last sharing session on Nov 30 and Dec 2.