Lecture 1 used 6ND on faith. Today we earn it. We build a transformer out of matrix multiplications, count its parameters exactly, count the FLOPs a token costs in each direction, and then count the bytes — which turn out to be the number that decides what hardware you need. No machine learning background is assumed; systems background is, and it is the right background, because every question we ask today is an accounting question. Then we run the same model forwards only, which is what serving is, and discover that the binding constraint moves from arithmetic to memory bandwidth — and that the batch size the hardware wants is ten times the batch size the memory allows. By the end you should be able to read a model configuration file and say what it costs to store, to train, to serve, and to serve well.

Where this sits

Lecture 1 drew the stack and did one calculation: serving compute overtakes training compute at T = 3D generated tokens. That calculation borrowed C ≈ 6ND without justification. Today supplies it, along with the reference model configuration that Part I reuses. The inference half of this same meeting — §2.11 onward — takes today's parameter count and today's FLOP rules, runs the model forwards only, and shows that the cost of serving is governed by bytes rather than by arithmetic. That is the fact the rest of the course is organized around, and §2.15's collision between a required batch size of 295 and an affordable one of 29 is the specific gap every serving lecture in Part II is trying to close.

Instructor notes — Timing plan

75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221). The Sep 2 period is shared with Lecture 1's course overview and with the meeting's inference half, so treat this full-period plan as the one to compress from.

Time	Segment	Notes
0–4	Recap and the reference 7B	Write the config on a side board and leave it up all semester. Do not derive yet.
4–9	2.1 Tokens and embeddings	Fast. This is the I/O boundary, not a topic.
9–20	2.2 What attention computes	The miniature with real numbers is the section. Do it by hand.
20–27	2.3 GEMM and not-GEMM	Classify, then the 0.19%-of-FLOPs / 15×-the-bytes contrast.
27–34	2.4 Parameter accounting	Derive N live, digit by digit. They will use it all semester.
34–43	2.5 Where the FLOPs go	2N from one multiply and one add. Then the crossover at 24,704.
43–52	2.6 The training loop and 6ND	Two backward GEMMs on the board is the whole factor of 3.
52–61	2.7 Four memory consumers	16 bytes/param, then activations dwarfing everything.
61–67	2.8 The budget, and the punchline	107.8 GB against 80. Say nothing for three seconds after.
67–72	2.9 What a framework does	Launch counting, then the PyTorch/TensorFlow argument.
72–75	2.10 Mixed precision	bf16 versus fp16, the master copy, done.

If running long: compress 2.9 to the launch-count arithmetic and one sentence on eager versus graph, and cut 2.10 to "bf16 has fp32's exponent range, which is why it needs no loss scaling." Protect 2.4, 2.6, and 2.8 — the parameter count, the factor of three, and the 107.8 GB. If those three land, the inference half works.

Learning objectives

By the end of this class you should be able to:

1. Compute the exact parameter count of a decoder-only transformer from L, d_model, d_ff, V, and the head configuration, and say which term dominates.
2. Derive the ≈2N FLOPs-per-token forward rule and the ≈4N backward rule, and explain the factor of two from the two backward GEMMs.
3. Classify every operation in a transformer block as GEMM or elementwise, and predict which ones are limited by arithmetic and which by memory traffic.
4. Compute the persistent training-memory footprint of a model under mixed-precision Adam in bytes per parameter, and separate it from weight memory.
5. Estimate activation memory for a given batch size and sequence length under a stated counting convention, and say whether a model fits on one 80 GB GPU.
6. Find the sequence length at which attention's own matrix multiplies rival the linear layers, for a given configuration.
7. Distinguish prefill from decode by the shape of their matrix multiplies, and predict which is compute-bound and which bandwidth-bound.
8. Compute a ridge point from a datasheet, classify a kernel against it by arithmetic intensity, and say what a generational bandwidth-versus-compute gap does to it.
9. Compute the batch-1 decode floor and the achieved utilization for any model and precision, and explain why that utilization is 1/ridge regardless of model size.
10. Compute KV-cache bytes per token and the maximum concurrency a card permits, and explain why grouped-query attention is a systems decision taken at pretraining time.
11. Convert a configuration into $/Mtok, joules per token, and a required batch size, and read Chinchilla from the serving side to say how big a heavily served model should be.

2.1 From text to numbers: tokens and embeddings

A language model does not see text. It sees a sequence of integers, each an index into a fixed vocabulary of V symbols. Tokenization — the rule that maps characters to those indices — is a preprocessing decision made once, before training, and it fixes the unit that every cost in this course is denominated in. For English prose a rough working figure is about four characters per token, which is an order-of-magnitude illustration rather than a constant; code, non-Latin scripts, and long numbers all tokenize much worse.

That matters more than it sounds. Latency is quoted per token. Throughput is tokens per second. Cost is dollars per million tokens. Context limits are token counts. If you change the tokenizer you change the denominator of every number in the system, and a comparison across two models with different tokenizers is not a comparison at all until you normalize.

The first layer of the network turns those integers into vectors. The embedding matrix has shape V × d_model, and the operation is a gather: look up row i, get a vector of d_model floating-point numbers. It is not a matrix multiply — it performs no arithmetic worth counting — but it does occupy V · d_model parameters, and it is read with terrible locality, since consecutive tokens in a batch index arbitrary rows.

At the other end the network produces, for each position, a vector of d_model numbers that has to become a probability over the whole vocabulary. That is a genuine matrix multiply against a d_model × V matrix, followed by a softmax. Some models tie the two matrices — the same weights used for input lookup and output projection — and some keep them separate. Our reference configuration keeps them separate, which costs an extra V · d_model parameters and matters when we count.

Everything between those two boundaries is the machine we spend the rest of the class accounting for.

Instructor notes

Minutes: 5. Board: Three boxes left to right: [ids] → [embed: V × d_model] → [... L blocks ...] → [unembed: d_model × V] → [logits]. Write V = 32000 and d_model = 4096 under the first and last. Ask the room: "Model A quotes 20 ms per token, model B quotes 18 ms. Which is faster?" Answer: unknown, until you know whether their tokenizers agree on how many tokens the same paragraph is. Expect confusion: Tokens get treated as words. Say: "A token is whatever the tokenizer says it is. Roughly four characters of English, and much worse on code."

2.2 What attention actually computes

Here is attention with no prerequisites. Every token in the sequence produces three vectors from its own representation: a query, a key, and a value. Think of the key as a label the token advertises, the value as the content it offers, and the query as the request the current token is making. To compute the output at position i, take that position's query, take the dot product against the key of every position up to and including i, turn those scores into weights that sum to one, and return the weighted average of the values.

That is a lookup table, made continuous. A hash table returns the one value whose key matches exactly. Attention returns a blend of all values, weighted by how well each key matches — and the blend is differentiable, which is why it can be learned. The dot product is the match score, and the softmax that normalizes the scores is what makes them weights.

Work one by hand. Take d_head = 2 and three tokens, and compute the output at position 3.

Two details that look like decoration and are not. The scaling by 1/√d_head exists because dot products of d_head-dimensional vectors grow with d_head, and large scores drive the softmax toward a one-hot distribution where gradients vanish. And the sum runs only up to position i, never past it — the causal mask — because a language model predicts the next token and must not see it. That mask is what makes generation sequential, and it is the reason the decode phase exists as a separate thing.

Real models run this many times in parallel. With h heads, each token produces h separate query/key/value triples of width d_head, attention runs independently per head, and the h outputs are concatenated back to width h · d_head = d_model. Different heads learn to attend to different things; from a systems point of view heads are just a batch dimension.

Now the reframing that matters for this course. Strip the interpretation and attention is: three matrix multiplies to produce Q, K and V from the input; one matrix multiply Q · Kᵀ to produce all scores at once; a softmax; one matrix multiply of the weights against V; and one more matrix multiply to project the result back to d_model. Five matrix multiplies and one softmax. The softmax is the only part that is not a GEMM, and §2.3 is about why that distinction is the one that decides performance.

Instructor notes

Minutes: 11. This is the section a student with no ML background either gets or does not. Board: Do the miniature by hand, digit by digit, including the exponentials. Do not display it finished. Then write the five matmuls in a column and circle "softmax" as the odd one out. Ask the room: "What happens to the output if I make all three scores equal?" Answer: a uniform average of the values — attention that has decided nothing. Useful for showing the softmax is doing selection. Expect confusion: Q, K, V get read as three different kinds of data. They are three linear projections of the same input vector, computed by three learned matrices. Say that sentence exactly. Common wrong answer: "The mask means we throw away half the work." In prefill the masked-out half is often computed and discarded anyway; the mask is a correctness device, not a performance one. Oct 5 makes it a performance one.

2.3 A transformer block is a small pile of matrix multiplies

A decoder-only transformer is L identical blocks stacked, each containing an attention sublayer and a feedforward (MLP) sublayer, each sublayer wrapped in a normalization and a residual add. Enumerate every operation in one block, for one token, and classify it.

Operation	GEMM?	FLOPs per token per layer
Normalization (×2)	no	2 · 4 · d_model = 32,768
Q, K, V projections	yes	3 · 2 · d_model² = 100,663,296
Attention scores Q·Kᵀ and weights·V	yes	4 · d_model · S = 16,384·S
Softmax over scores	no	≈ 5 · h · S = 160·S
Output projection	yes	2 · d_model² = 33,554,432
Gate and up projections	yes	2 · 2 · d_model · d_ff = 180,355,072
Activation function and gating multiply	no	≈ 5 · d_ff = 55,040
Down projection	yes	2 · d_model · d_ff = 90,177,536
Residual adds (×2)	no	2 · d_model = 8,192

The GEMM rows sum to 404,750,336 FLOPs per token per layer, plus the sequence-dependent attention term. The non-GEMM rows, at S = 4096, sum to about 751,000 — 0.19% of the layer's arithmetic.

So the elementwise work is free. Except it is not, and the reason is bytes. A normalization reads d_model values and writes d_model values while doing about four operations per value: at bf16 that is 4 · 4096 = 16,384 FLOPs against 2 · 4096 · 2 = 16,384 bytes, an arithmetic intensity of exactly 1 FLOP per byte. The inference half will give you the H100's ridge point of 295 FLOP/byte; a kernel at intensity 1 runs at 1/295 of the machine's arithmetic peak, which is to say it is doing nothing but moving memory.

Count the traffic for a realistic step. Under a conservative accounting — each elementwise operation reads its inputs and writes its output once, nothing fused — the non-GEMM operations in one layer touch about 192,000 bytes per token. At a training microbatch of 32,768 tokens that is 6.29 GB per layer, against 0.405 GB of weights in that same layer. The cheap operations move fifteen times more bytes than the expensive ones. Across 32 layers it is 201 GB of traffic, about 60 ms on an H100's 3.35 TB/s, against roughly 1.1 s of forward-pass arithmetic at 40% of dense BF16 peak — five percent of the forward pass spent on 0.19% of the FLOPs.

That asymmetry is the entire reason kernel fusion exists. If the normalization, the activation, and the residual add can be executed while the data is still in registers or shared memory, the intermediate round trips to HBM disappear and the elementwise work costs almost nothing. How that is done — tiling, fusion, and the IO-aware attention kernel that applies the same idea to the softmax — is Sep 30 and Oct 5. Today the point is only that you can predict which operations will benefit, before writing any code, from a FLOP count and a byte count.

Instructor notes

Minutes: 7. Board: Two columns, "GEMM" and "not GEMM". Fill them from the block diagram, then write "0.19% of FLOPs" under the right column and "15× the bytes" under it. The gap between those two lines is the section. Ask the room: "Which of these operations would you optimize first?" Most say the big matmuls. Then show the byte count. Expect confusion: FLOP count is treated as a proxy for time. Say: "Time is max of arithmetic and traffic, not arithmetic alone. A kernel at 1 FLOP/byte is a memcpy with opinions." If short on time: Keep the intensity-of-1 calculation and the 15× line; drop the 5%-of-forward estimate.

2.4 Parameter accounting

Fix a configuration and use it for the rest of Part I. Call it the reference 7B: L = 32 layers, d_model = 4096, h = 32 attention heads, d_head = 128, d_ff = 11008, V = 32000, no grouped-query attention, untied input and output embeddings, no biases. It is deliberately ordinary — an open 7B-class decoder — and the inference half reuses it without change.

Attention contributes four square matrices per layer: Q, K, V, and the output projection, each d_model × d_model. Note h · d_head = 32 · 128 = 4096 = d_model, so the per-head split costs nothing extra. The MLP contributes three matrices in the gated style now standard: a gate and an up projection, each d_model × d_ff, and a down projection d_ff × d_model.

Three observations about where the mass sits. The MLP is 135,266,304 of the 202,375,168 per-layer parameters — two-thirds of the model is the feedforward network, not attention, which surprises people whose mental model of a transformer is the attention diagram. The embeddings are 262,144,000 of 6,738,149,376, or 3.9%; they are negligible here and would not be for a small model with a large vocabulary. And the normalization parameters — one vector of d_model per normalization, 65 of them — total 266,240, about 0.004% of N. We omit them and say so; that is the right kind of omission because it is four orders of magnitude below the leading term.

Now the sensitivity. Per-layer parameters are d_model · (4·d_model + 3·d_ff), and with d_ff ≈ 2.6875 · d_model that is 12.06 · d_model², so the body of the model is L · 12.06 · d_model². Parameters grow linearly in depth and quadratically in width. Doubling L to 64 gives 12.95B in the body; doubling d_model to 8192 gives 25.9B. If you want a bigger model, width is the aggressive lever.

Width is also the cheaper lever per parameter, for two systems reasons. Wider matrices mean larger GEMMs, which reach a higher fraction of peak; deeper models mean more sequential kernel launches, more layers on the critical path, and under tensor parallelism more collectives per token. And the KV cache — the inference half's subject — scales as L · n_kv · d_head, so quadrupling parameters by doubling d_model doubles the cache, while quadrupling them by quadrupling L quadruples it. Depth is not free, and the reasons it is not free are almost all systems reasons. Depth still buys something real about what the model can represent, which is why nobody builds a two-layer 7B, but the trade is being made against a real cost.

Instructor notes

Minutes: 7. Board: Derive all five lines live. Make them do 4 · 4096² in their heads and check against 67,108,864. Box N = 6,738,149,376 and 13.5 GB — both go up on the permanent side board. Ask the room: "Attention or MLP — which has more parameters?" Most say attention. It is two to one the other way. Expect confusion: "7B" is read as an exact 7,000,000,000. Say: "It is a marketing rounding of 6.74 billion. Every number we derive this semester uses 6.738, not 7." If short on time: The count must survive intact. Cut the depth-versus-width paragraph to its first sentence.

2.5 Where the FLOPs go

Take one linear layer, Y = XW, with W of shape d_in × d_out, and one token of input. Each output element is a dot product of length d_in: d_in multiplies and d_in adds. There are d_out outputs, so the cost is 2 · d_in · d_out FLOPs — and d_in · d_out is exactly the number of parameters in W. Every parameter that participates in a matrix multiply costs two FLOPs per token: one multiply and one add.

For the reference 7B that is 2 · 6.738e9 = 13.48 GFLOP per token, forward. Two honesty notes. The rule slightly over-counts, because the embedding lookup is a gather rather than a matmul, and treating its 131,072,000 parameters as if they multiplied adds 1.9% that is not there. It also under-counts, because it omits the attention score and value matmuls, whose cost depends on parameters not at all — and that omission is the interesting one.

Attention's own matmuls scale with sequence length. For a token at position S, per layer, the scores cost 2 · h · d_head · S FLOPs and the weighted sum of values costs the same, giving 4 · d_model · S = 16,384 · S FLOPs per token per layer. There are no parameters in that expression. It is pure data movement through arithmetic, and it grows without bound as the context does, which is why the linear-layer accounting that dominates at short context stops being the whole story at long context.

Find where they meet. Per layer per token, the linear layers cost 2 · 202,375,168 = 404,750,336 FLOPs and attention costs 16,384 · S. Setting them equal:

At about 24,700 tokens of context, attention's matmuls cost as much as every weight matrix in the layer. Below that they are a correction; above it they are the dominant term, and because the cost is linear in S per token — quadratic in S for a whole prefill — it keeps growing. At S = 4096 attention is 16.6% of the layer's arithmetic. At S = 100,000 it is four times the linear layers. This is the precise sense in which long context stops being free, and it is worth noticing that it happens at a context length products now advertise routinely.

The closed form is worth keeping: the crossover is the per-layer parameter count divided by 2 · d_model, which for the standard d_ff = 4·d_model shape works out to exactly 2^15 = 32,768. Fatter MLPs push the crossover out; wider models push it out proportionally to their parameter growth. Note also what the crossover is not: it says nothing about memory. The bytes attention consumes — the KV cache — start hurting long before 24,704 tokens, and that is the inference half's KV-cache accounting.

Instructor notes

Minutes: 9. Board: Y = XW, one row of X, one column of W, count one multiply and one add per weight. Then 2N boxed. Then 4·d_model·S beside it, and solve for S in front of them. Ask the room: "How many FLOPs does the attention score matmul spend per parameter?" It has no parameters. That is the point, and the silence before someone says it is productive. Expect confusion: "Attention is quadratic" is remembered as a claim about the whole model. Say: "Quadratic in S for one term, out of a model whose other terms are linear. Which one wins depends on S, and here the answer is about 25,000." Common wrong answer: Students compute the crossover using 2N for the whole model instead of per layer. It lands at 25,704 — close enough that they will not notice the error, so name it.

2.6 The training loop, and where the factor of three comes from

Training is a loop: run the model forward on a batch, compare its predictions to the actual next tokens, compute how each parameter should change, change them, repeat. In PyTorch it is about ten lines.

for batch in loader:
    ids = batch["input_ids"].to(dev)      # H2D copy: B·S int64, trivial bytes
    logits = model(ids)                   # ≈2N FLOP/token; allocates the activation tape
    loss = cross_entropy(                 # logits are B·S·V — 2.10 GB at bf16, 4.19 GB at fp32
        logits[:, :-1].reshape(-1, V),
        ids[:, 1:].reshape(-1))
    loss.backward()                       # ≈4N FLOP/token; fills .grad, frees the tape
    optimizer.step()                      # no matmuls; ~28 bytes/param of traffic → 189 GB
    optimizer.zero_grad(set_to_none=True) # releases 2 bytes/param

Read that as a resource trace rather than as code. The forward pass costs 2N FLOPs per token and, more importantly, allocates every intermediate tensor the backward pass will need — that allocation is §2.7's subject and it is the line that decides your batch size. The loss line materializes a tensor of shape B · S · V; at a 32,768-token microbatch and V = 32000 that is 2.10 GB in bf16 and 4.19 GB if the softmax is done in fp32, which is routinely the single largest tensor in the step and routinely a surprise. The backward pass costs about twice the forward. The optimizer step does no matrix multiplies at all but streams roughly 28 bytes per parameter — 189 GB for the reference 7B, about 56 ms on an H100 — which is invisible at large batch and is not at small batch.

Now the factor of two. Consider Y = XW again, and suppose the backward pass has arrived carrying dY, the gradient of the loss with respect to Y. Two things are needed. The gradient with respect to the input, so the chain rule can continue down the network:

and the gradient with respect to the weights, which is what the optimizer will consume:

Both are matrix multiplies of the same shape class as the forward's single one, each costing 2 · d_in · d_out FLOPs per token. The backward pass does two GEMMs where the forward did one. That is the entire factor: ≈2N FLOPs per token forward, ≈4N backward, ≈6N for the pair.

That is Lecture 1's rule, and now it is derived rather than asserted. Lecture 1 used it to show that cumulative serving compute overtakes training compute at T = 3D generated tokens — a token generated costs 2N while a token trained on costs 6N, so three of the former equal one of the latter — and that result stands; we do not re-derive it here.

Where 6ND is wrong, and in which direction. It counts only the parameter-bearing matmuls, so it omits the attention score and value terms of §2.5, which matters above S ≈ 25,000 and is a rounding error below. It ignores activation recomputation, which is a deliberate trade of extra forward FLOPs for less memory and can add 30% or more to the true cost. It ignores everything a real project spends outside the successful run: failed runs, restarts from checkpoints, hyperparameter search, data pipeline work. And it says nothing about utilization — 6ND is a FLOP count, and converting it to wall clock requires an efficiency assumption that is usually somewhere between 30% and 50% and that you should never take from a paper without asking how it was measured. Every one of those corrections pushes the real cost up, none down.

Instructor notes

Minutes: 9. Board: The two backward equations, one above the other, with "same shape as the forward" written beside each. Then 2N + 4N = 6N and box C ≈ 6ND. This is the most reusable ten seconds of the lecture. Ask the room: "Why exactly two, and not three or one and a half?" Because there are exactly two things downstream needs: the gradient flowing further back, and the gradient of this layer's own weights. Expect confusion: Backward is imagined as a second forward pass in reverse. Say: "It is a different computation with a different FLOP count, and the count is two GEMMs per forward GEMM." If short on time: The two equations and the boxed 6ND. The caveat list can be assigned as reading.

2.7 Memory: four consumers, and the one that surprises people

Ask a student how much memory it takes to train a 7B model and most say fourteen gigabytes, because the weights are 13.5 GB. That answer is wrong by roughly an order of magnitude, and the gap is the most consequential thing in this lecture.

Four things occupy memory during training. Weights, the parameters themselves. Gradients, one number per parameter, produced by the backward pass. Optimizer state, which for Adam is two additional numbers per parameter — a running mean of the gradient and a running mean of its square — plus, under mixed precision, a full-precision master copy of the weights. And activations, the intermediate tensors saved during the forward pass because the backward pass needs them.

The first three are proportional to N and are the same every step. Under standard mixed-precision Adam:

Consumer	bytes per parameter
bf16 weights (used in the matmuls)	2
fp32 master weights	4
Adam first moment, fp32	4
Adam second moment, fp32	4
persistent subtotal	14
bf16 gradients	2
working figure	16

The subtotal is 2 + 4 + 4 + 4 = 14 bytes per parameter. Whether you carry 14 or 16 depends on whether your framework frees the gradient buffer inside the optimizer step; we carry 16 bytes per parameter and state that we are counting a bf16 gradient. Be explicit about the convention whenever you quote a figure like this, because the classic error in this area is not arithmetic — it is comparing someone else's 14 to your 16 without noticing that they counted different things.

For the reference 7B: 6,738,149,376 · 16 = 107,810,390,016 bytes = 107.8 GB, of which the weights are 13.5 GB. Weight memory is 12.5% of persistent training memory. Turn the same arithmetic around and the largest model whose persistent state alone fits in 80 GB is 80e9 ÷ 16 = 5 billion parameters — before a single activation, before a single token of data.

Activations are the fourth consumer and they behave differently: they scale with the number of tokens in the microbatch, not with N. Counting them requires a convention, because frameworks disagree about what they keep. Ours, stated so it can be attacked: per layer per token, in bf16, we save the normalized input to the QKV projections, Q, K and V, the attention output before the output projection, the normalized MLP input, the gate and up projections, and the gated product feeding the down projection.

This is one reasonable convention and not the only one. A framework with attention fused end to end saves less; one that keeps a separate copy of every normalization input and output saves more; one that materializes the S × S attention matrix saves catastrophically more. Treat 3.69 MB per token as the right order of magnitude for this configuration and re-derive it for whatever stack you actually measure.

The practical consequence is that activation memory is the term you control, because B and S are yours to choose and N is not. Microbatching splits the batch you want into pieces small enough to fit; gradient accumulation runs several microbatches, summing gradients into the same buffers, and calls the optimizer once, so the optimizer sees the large batch the training recipe asked for while memory only ever holds one microbatch of activations. The cost is that the weights and optimizer state are read once per accumulation step rather than once per large batch, which at small microbatch sizes starts to matter. Activation recomputation goes further: discard most activations during the forward pass and recompute them from checkpoints during the backward, trading roughly an extra forward pass of FLOPs for a large reduction in memory. That trade, and the parallelism strategies that make the persistent 107.8 GB somebody else's problem, are optional content rather than lectures.

Instructor notes

Minutes: 9. Board: The bytes-per-parameter table, built one row at a time with the question "what else does Adam need?" between rows. Write 107.8 GB and 13.5 GB one above the other and draw the ratio. Ask the room: "How much memory to train a 7B model?" Take the answers before showing the table. Almost everyone says 14 GB. Expect confusion: The master weights are read as a redundant copy that a careful implementation could drop. Say: "It is the only copy with enough mantissa bits to accumulate a small update. §2.10 shows the arithmetic." Common wrong answer: "Activations are small because each one is small." Each one is a few kilobytes and there are L per token and tens of thousands of tokens.

2.8 A worked memory budget, and the punchline

Put it together and answer the question a practitioner actually asks: does the reference 7B train on one 80 GB GPU?

That last clause is the punchline, so sit with it. Activations are not the problem here. The persistent state alone, 107.8 GB, exceeds the card by 35%, and no batch size, no sequence length, no recomputation strategy, and no fusion changes that number by a byte. It is N · 16, and the only levers on it are a smaller model, a smaller optimizer, or more GPUs.

The choice the field made is more GPUs, and the observation that gets you there is that the 94 GB of gradients and optimizer state is not needed in full on every device. Each of eight GPUs could hold an eighth of the optimizer state and an eighth of the gradients, exchanging what is needed when it is needed — trading interconnect bandwidth for memory capacity. That is ZeRO's idea, and together with the orthogonal question of splitting the model itself across devices (Megatron-LM) it is the optional-content track on the course page, not a lecture. Do not go looking for those mechanisms today. Take away only that distributed training is not a scaling optimization; it is a correctness requirement, imposed by a single division. Nobody chose to make training distributed. Sixteen bytes per parameter did.

One arithmetic check that is worth running yourself. If persistent state is 16 bytes per parameter, then a 70B model needs 1.12 TB before activations — fourteen 80 GB cards' worth of memory, held by a model whose weights are 140 GB. The ratio never improves with scale; it is a constant.

Instructor notes

Minutes: 6. Board: Stack the five persistent rows, sum them, write "80" underneath, and draw the line. Say nothing for three seconds. Ask the room: "What do you cut?" Let them propose smaller batches, then point at the subtotal. The realization that batch size is irrelevant here is the moment. Expect confusion: Students assume gradient checkpointing or a smaller sequence length rescues this. Say: "Those touch the 15 GB, not the 108." If short on time: Nothing in this section is cuttable. Cut §2.9 instead.

2.9 What a framework actually does for you

You wrote ten lines in §2.6 and never described the backward pass. Something computed it, and that something is worth understanding, because its design decisions show up as performance.

The mechanism is autograd as a tape. As the forward pass executes, the framework records each operation and the tensors it consumed onto a directed graph. Calling .backward() walks that graph in reverse, and at each node applies the local derivative rule — the two GEMMs of §2.6 for a linear layer, and an analogous rule for every other operation — chaining gradients from the loss back to every parameter. Two consequences follow directly. The tape holds references to the forward's intermediate tensors, which is why activations are a memory consumer at all rather than transient scratch. And the tape is built from the control flow that actually ran, so a Python if in the model is recorded as whichever branch it took.

The second thing a framework does is launch kernels, and launches are not free. Each one costs the host CPU some microseconds of dispatch work to enqueue onto the GPU stream. Treat that as order 5 µs in eager mode — a clearly-flagged order of magnitude, not a measurement, and one you should measure on your own stack before relying on it.

Count them. A reasonable eager implementation of one transformer layer forward issues on the order of 15 kernels: two normalizations, three projection GEMMs, a rotary embedding application to Q and K, a fused attention call, an output GEMM, two residual adds, two MLP GEMMs, an activation, a gating multiply, and a down GEMM. Backward roughly doubles that, so call it 45 per layer per step. Across 32 layers that is 1,440, plus perhaps 60 more for embeddings, the final normalization, the logits GEMM, and the loss — about 1,500 launches per training step, before the optimizer. A foreach-style fused Adam adds a handful; a naive per-tensor implementation issuing ten elementwise kernels for each of ~291 parameter tensors adds nearly 3,000.

Now compare against the step's compute. At B = 8, S = 4096 the step processes 32,768 tokens, so C = 6 · 6.738e9 · 32,768 = 1.32e15 FLOPs, and at 40% of the H100's 989 TFLOP/s dense BF16 peak that is 3.35 s. Against 3.35 seconds, 1,500 launches at 5 µs is 7.5 ms — 0.2%, invisible.

Shrink the step and the picture inverts. Setting 1,500 · 5 µs equal to 6 · N · D_step ÷ 3.956e14 gives D_step ≈ 75 tokens. Below roughly 75 tokens per step, this model spends more time launching kernels than executing them. That regime is not hypothetical: it is exactly the decode phase the inference half prices, where each step processes one token per sequence. A forward-only eager pass over 32 layers is on the order of 490 launches, about 2.45 ms of dispatch, against the inference half's 4.0 ms memory-bandwidth floor for the same step. Launch overhead alone can be half the budget of a decode step. The answers — CUDA graphs, which record a launch sequence once and replay it as a unit, and compilation, which fuses many small kernels into few large ones — are Sep 30 and Oct 7 material.

Which brings us to the two assigned system papers, and the argument between them. TensorFlow asked the user to declare the computation as a graph, then compiled and optimized that graph before running it. Given a whole graph a system can fuse operations, plan memory, place work across devices, and eliminate exactly the launch overhead just counted. PyTorch executed operations immediately as Python called them, giving up the whole-program view in exchange for a program you can debug with a print statement and a stack trace and a Python if.

PyTorch won, and it is worth being precise about why, because the reason is a systems lesson rather than an ML one. It did not win on throughput; the graph-first design had every structural advantage on throughput. It won because researchers iterate, and the cost of iteration — write, run, misread the error, fix, run again — dominated the cost of execution for the work that was actually being done. The system optimized for the workload's real bottleneck, which was the human. The PyTorch paper is candid about this being a deliberate trade rather than a free lunch, and its engineering sections are about how much performance you can recover inside an eager design.

The postscript is that graphs came back. torch.compile, CUDA graphs, and the tracing compilers now standard in serving stacks all recover the whole-program view — but as an opt-in applied to an already-working eager program, rather than as the price of entry. The winning design was eager first, graph second, and that ordering is the thing to remember.

Instructor notes

Minutes: 5. Board: 1,500 launches × 5 µs = 7.5 ms above step compute = 3.35 s, then erase the second and write 75 tokens under it. The inversion is the section. Ask the room: "TensorFlow could fuse across the whole graph and PyTorch could not. Why did PyTorch win?" Push past "it was easier" to "iteration speed was the binding constraint, and they optimized the right resource." Expect confusion: Eager versus graph is heard as a settled historical question. Say: "It settled as eager-first with graphs bolted back on. Both halves of that sentence matter." If short on time: Keep the 75-token crossover and one sentence on the design bet.

2.10 Mixed precision, only as far as the accounting needs

We have been assuming b = 2 bytes per parameter for the weights that participate in matmuls, and b = 4 for the master copy and the optimizer moments. Two questions follow: which 16-bit format, and why keep a 32-bit copy at all.

Both fp16 and bf16 use 16 bits and split them differently. fp16 spends 5 bits on the exponent and 10 on the mantissa, giving good precision over a narrow range that tops out around 65,504 and, more dangerously, underflows to zero for small values. bf16 spends 8 bits on the exponent — the same as fp32 — and 7 on the mantissa, so it covers fp32's full dynamic range with about three significant decimal digits. Gradients in a deep transformer span many orders of magnitude, and in fp16 the small ones silently become zero. The historical workaround is loss scaling: multiply the loss by a large constant before the backward pass to lift gradients into fp16's representable range, unscale before the optimizer step, and back off when an overflow is detected. It works, it is fiddly, and it is a source of silent divergence. bf16 needs none of it, because it never had the range problem, and that is essentially the whole reason it displaced fp16 for training.

The master copy answers the second question. bf16 has 8 bits of mantissa precision, so the gap between representable numbers near a value w is about w · 2⁻⁸ ≈ 0.39% of w. An Adam update has magnitude roughly the learning rate, since the moment ratio is normalized to order one. Early in training, with a learning rate of 3e-4 against weights of order 0.02, each update is about 1.5% of the weight — comfortably above the rounding threshold. But learning-rate schedules decay by one to two orders of magnitude, and a late update of 0.015% of the weight, applied to a bf16 number, rounds to no change at all. Every step of the tail of training would be a no-op. The fp32 master copy, with relative resolution near 6e-8, accumulates those updates; the bf16 copy used in the matmuls is derived from it after each step. Four bytes per parameter is what it costs to keep the last third of a training run from doing nothing.

This is worth separating cleanly from a topic that sounds identical. Quantizing a model for inference — Nov 9 — takes a finished set of weights and asks how few bits can represent them while the outputs stay acceptable. There is no accumulation of small updates, no optimizer, and no schedule; the only question is output quality. Training precision is about whether a number can absorb a small change, inference precision is about whether a number is close enough. They share a datatype table and nothing else.

The last observation is the one that pays off in the inference half. Precision is a knob on bytes, and bytes have shown up in every section today: 16 bytes per parameter of persistent state, 3.69 MB per token of activations, 189 GB of optimizer traffic, 201 GB of elementwise traffic. In the inference half the same knob turns the weight-streaming floor and the KV-cache footprint. Once you have internalized that almost every quantity in this course is a byte count divided by a bandwidth, the rest of the semester is bookkeeping with high stakes.

Instructor notes

Minutes: 3. Board: bf16: 8 exponent bits, 7 mantissa over fp16: 5 exponent, 10 mantissa, then 2⁻⁸ ≈ 0.39% and lr/|w| ≈ 0.015% late in training. Two lines, then stop. Ask the room: "If bf16 is less precise than fp16, why did it win?" Range beats precision when your gradients span ten orders of magnitude. Expect confusion: Mixed precision is assumed to halve training memory. It does not touch the 16 bytes per parameter — Exercise 2 makes them prove it. It halves activations and raises arithmetic throughput by 2× against TF32 tensor and 15× against non-tensor FP32.

2.11 Running it forwards: prefill and decode are two different machines

Everything so far ran the model forwards and backwards. Serving runs it forwards only, and the moment you do, it splits into two phases with completely different performance characteristics. Most mistakes in this course come from reasoning about one while thinking about the other.

Prefill takes the prompt — S tokens — and runs one forward pass over all of them, producing the KV cache for those positions and the first output token. You will see it called the prompt phase or context encoding; we say prefill. Because all S tokens pass through the same weight matrices together, prefill is a sequence of large matrix-matrix multiplications: it looks like §2.6's forward pass and behaves like it.

Decode produces one token at a time — take the token just emitted, run it through the whole network, attend over the KV cache, emit the next. The synonyms are generation phase and incremental decoding. A decode step for one sequence is a sequence of matrix-vector multiplications, one token of activations against every weight matrix in the model, and it is serial: token i+1 cannot start before token i exists.

The claim the rest of this note unpacks: prefill is compute-bound because it does plenty of arithmetic per byte of weight it reads, and decode is memory-bandwidth-bound because at small batch size it does almost none. On one H100 the same model in the same precision runs three orders of magnitude apart in achieved FLOP/s depending only on which phase it is in. A request with a 4,000-token prompt that generates 200 tokens performs 201 passes over all 6.74 billion parameters: one prefill and two hundred decodes. The API hides the loop; the GPU does not.

Instructor notes

Minutes: 5. This is the hinge of the meeting — the training half ends, the serving half begins. Board: Two boxes — "prefill: S tokens → 1 pass" and "decode: 1 token → 1 pass, ×output length". Write "matrix × matrix" under the first, "matrix × vector" under the second. Leave room underneath for §2.12's ridge points. Ask the room: "A request has a 4,000-token prompt and generates 200 tokens. How many forward passes over the full network happen?" Answer: 201. Wait for it. Expect confusion: Students who have only used a chat API think generation is one function call. Say: "One HTTP request, 201 passes over 6.7 billion parameters."

2.12 Arithmetic intensity and the roofline

Arithmetic intensity is FLOPs performed per byte moved to or from memory. It is a property of an algorithm on a machine, not of either alone, and it is the most useful single number in this course. §2.3 already computed one: a normalization sits at exactly 1 FLOP per byte.

The roofline model (Williams, Waterman, and Patterson, 2009) turns that ratio into a prediction. Plot attainable throughput against intensity on log-log axes: at low intensity the memory system limits you and the ceiling is a diagonal, intensity × bandwidth; at high intensity the arithmetic units limit you and the ceiling is flat at peak FLOP/s.

The lines meet at the ridge point, where a kernel just saturates both resources; below it you are bandwidth-bound, above it compute-bound. Ridge point = peak FLOP/s ÷ bandwidth: 153 FLOP/byte on an A100 80GB SXM (312e12 ÷ 2.039e12) and 295 FLOP/byte on an H100 SXM (989e12 ÷ 3.35e12). Both are dense BF16 tensor-core peaks — the :::tip at the end of §2.10 is why.

Read the two against each other, because the trend is the whole story. From A100 to H100, dense BF16 compute rose 989/312 ≈ 3.2× while bandwidth rose 3,350/2,039 ≈ 1.6×, so the ridge point rose 295/153 ≈ 1.9×. Each generation demands roughly twice the arithmetic per byte before it runs at full speed, so a workload with fixed low intensity gets the bandwidth improvement and nothing else. The bandwidth wall tightens with every generation, which is why Part II is as long as it is.

Instructor notes

Minutes: 7. Board: Draw the axes and both lines before naming either bound, then ask which region a matrix-vector product lives in. Derive both ridge points live. Write "3.2× compute, 1.6× bandwidth" and let them compute 1.9×. Ask the room: "The H100 has 3.2× the FLOP/s of an A100. If your kernel was bandwidth-bound on the A100, what speedup do you get?" Answer: 1.6×. This is where the roofline stops being decorative. Expect confusion: Intensity gets confused with FLOP count. Say: "Intensity is a ratio. A huge kernel and a tiny kernel can have identical intensity and identical efficiency."

2.13 The decode floor: one stream, one GPU

At batch size 1 a decode step must read essentially all of §2.4's 13.5 GB: each weight matrix is used once, against one token of activations, then not touched again until the next token. There is no reuse to exploit, so the step cannot finish faster than the weights can be streamed.

The floor is linear in b, and that deserves a table rather than a sentence, because halving the bytes per weight is the single most-quoted fact in quantization. Same model, same arithmetic — N · b bytes divided by bandwidth:

Precision	b	Weights	H100 floor	A100 floor
bf16	2	13.5 GB	4.0 ms → 249 tok/s	6.6 ms → 151 tok/s
fp8	1	6.74 GB	2.0 ms → 497 tok/s	3.3 ms → 303 tok/s
int4	0.5	3.37 GB	1.0 ms → 994 tok/s	1.65 ms → 605 tok/s

Two points, and only two. First, the floor is a statement about bytes moved, not about datatypes the tensor cores support: the A100 has no fp8 arithmetic at all, and weight-only quantized models typically dequantize and do their math in bf16 anyway — yet the fp8 and int4 rows hold on both GPUs regardless, because halving b halves the bytes and therefore halves the floor, full stop. Second, whether the model is still worth serving at those precisions is a quality question, and it belongs to Nov 9, not to today.

Now ask what the machine does while it waits. Decode costs ≈2N FLOPs per token per sequence, so 13.5 GFLOP; divide by 4.02 ms for 3.35 TFLOP/s against 989 TFLOP/s of peak — 0.34%. That 3.35 is exactly the bandwidth in TB/s times one, which is no coincidence: at bf16, batch-1 intensity is 2N FLOPs ÷ N·2 bytes = exactly 1 FLOP/byte, independent of model size, and 1/295 = 0.34%. On the A100 it is 1/153 = 0.65% — the older, cheaper GPU wastes proportionally less of itself.

Scale up honestly. A 70B-class model in bf16 needs 140 GB of weights, which does not fit in 80 GB, and quoting a single-GPU figure for a model that does not fit on one GPU is the standard way to get this wrong. Take tensor parallelism across two H100s: each streams its own 70 GB, aggregate bandwidth 6,700 GB/s, floor 140 ÷ 6,700 = 20.9 ms per token, about 48 tokens/s, achieving 2 · 70e9 ÷ 0.0209 = 6.7 TFLOP/s against 1,978 installed — 0.34% again, because batch-1 utilization is always 1 over the ridge point. Eight-bit quantization onto one GPU reaches the same floor from half the bytes but leaves only about 10 GB for the KV cache, and the fp8 peak doubles too, so the ratio is unchanged: quantization buys latency and capacity, not efficiency. The two-GPU floor is optimistic besides, since tensor parallelism adds a collective after each attention and feedforward block — 64 small all-reduces per token whose fixed latency lands on a 21 ms critical path. That is §3 of the required paper.

Instructor notes

Minutes: 8. Board: Divide 13.5 by 3.35 in front of them — do not display 4.0 ms. Then write "3.35 TFLOP/s out of 989" and wait for the reaction before saying 0.34%. The reaction is the point. Ask the room: "You have a 4 ms floor and want 100 tokens/s from one stream. What can you change?" Push to "move fewer bytes" (quantization, Nov 9) or "read the weights fewer times per token" (speculative decoding, Nov 11). Expect confusion: Many assume the GPU is compute-limited because "GPUs are for math". Say: "At batch one the GPU spends 99.7% of the step waiting for memory." Common wrong answer: "Buy a bigger GPU." The ridge point rises faster than bandwidth, so single-stream utilization gets worse.

2.14 Batching is the only lever, and it has a ceiling

Raise the intensity. The weights are read once per step regardless, so if the step processes B sequences the same bytes serve B times the arithmetic, and the matrix-vector products become matrix-matrix products with a short inner dimension.

Set that equal to the ridge point and read off the batch size required: B ≈ 153 on an A100, B ≈ 295 on an H100. Below those you are paying for arithmetic units you are not using. The threshold is a property of the hardware alone — N cancels — and the H100 needs nearly twice the concurrency to earn its price.

Prefill is the same phenomenon in different clothing, putting S tokens through one pass over the weights. For the reference 7B at S = 4096 the weight GEMMs cost 2 · 6.74e9 · 4096 = 55.2 TFLOP, the quadratic attention adds 4 · 32 · 32 · 128 · 4096² = 8.8 TFLOP for 64.0 TFLOP total, against 13.5 GB of weights plus 2.1 GB of KV written = 15.6 GB. Intensity ≈ 64.0e12 ÷ 15.6e9 ≈ 4,100 FLOP/byte, fourteen times past the ridge point. One quantity governs both phases — tokens per pass over the weights — and prefill has thousands of it while decode at batch 1 has one.

Instructor notes

Minutes: 5. Board: intensity ≈ B, then the two ridge points, then circle 295. Ask for the number before giving it. Ask the room: "Does the required batch size depend on the model?" No — N cancels. Worth the ten seconds; it surprises people. Expect confusion: Batch size gets conflated with concurrent users. In decode the batch is the set of sequences generating in that step, and its composition changes every step in a real server — Oct 7.

2.15 The KV cache, and why the ceiling is out of reach

Batching costs memory, and the memory it costs is the KV cache: every sequence keeps its own keys and values for every position it has seen, in every layer. This is not an optimization you can decline — without it, each decode step would recompute attention over the whole prefix, turning linear generation into quadratic.

The leading 2 is K and V. The reference 7B has no grouped-query attention, so n_kv = 32.

That factor of ten is the central fact of modern LLM serving: the hardware demands a batch size the memory will not permit. Everything on the serving side of Part II responds to this gap. And it gets worse first — at S = 32,768 each sequence needs 524,288 · 32768 = 17.2 GB, so three sequences fit. Long context is a memory-capacity problem before it is a modelling problem.

Grouped-query attention is the architectural response, and Exercise 1 already counted its parameters: several query heads share one key/value head, cutting n_kv below the query-head count. With n_kv = 8 the 32 query heads and the FLOPs barely change, but the cache shrinks 4× — 128 KiB per token, 0.537 GB per sequence at 4K, 62.5 ÷ 0.537 ≈ 116 sequences. Compare the two at their maxima, both filling the same memory:

Config	KV/token	KV/seq at 4K	Max B	Bytes/step	TPOT	Aggregate
MHA, n_kv = 32	512 KiB	2.15 GB	29	75.8 GB	22.6 ms	~1,280 tok/s
GQA, n_kv = 8	128 KiB	0.54 GB	116	75.8 GB	22.6 ms	~5,130 tok/s

Same GPU, same bytes per step, same per-token latency, four times the throughput — a pretraining-time decision about the weights, taken for a purely systems reason.

A sharper point hides in that table. At B = 29 the step moves 75.8 GB, only 13.5 of it weights, so batching did not lift the intensity to 29: the true figure is 29 · 15.6 GFLOP ÷ 75.8 GB ≈ 6 FLOP/byte, because KV traffic grows with B exactly as fast as the arithmetic does. Attention over the cache alone sits at exactly 1 FLOP/byte at bf16 with MHA regardless of B, S, or head count — the FLOPs and the bytes are the same expression — and never benefits from batching, because no two sequences share a cache. Grouped-query attention lifts that term to 4 and the whole step to about 24. Both are far below 295: batching helps enormously and still leaves the machine mostly idle.

The Part II answers, named only, each with its date. Paged allocation stops each cache being over-provisioned to its maximum length (Sep 28, Oct 7); prefix reuse stops recomputing the cache for prompt prefixes many requests share (Nov 4); cache compression and quantization shrink b for the cache itself (Oct 28, Nov 9); disaggregation stops prefill and decode fighting over the same card (Oct 14). The final project — 12% of the grade, announced Oct 26 — is where you could point one of these at a workload of your own, provided you can show that nothing you can already download does the job.

Instructor notes

Minutes: 10. Protect this budget; it is the most important arithmetic of the serving half. Board: Derive 512 KiB/token digit by digit — they will use it all semester. Then 2.15 GB, 62.5, 29. Write "29" and "295" side by side and circle the gap. GQA row next; the 6-FLOP/byte correction last. Ask the room: "Where did the memory go?" Make them say it: 13.5 GB of weights and 62 GB of cache, for 29 users. Expect confusion: The cache is treated as optional. Say: "Without it, generating token 4,000 recomputes attention over 4,000 positions. You cannot turn it off, only make it smaller." Common wrong answer: "Then batch 295 with a shorter context." At 512 KiB/token that allows about 400 tokens each — show it, it kills the idea and motivates Sep 28. If short on time: The 6-FLOP/byte correction can go; 29-versus-295 and the GQA row cannot.

2.16 Latency, cost, and the scaling laws read from the serving side

"Latency" is not a number for a streaming system, and the four metrics that replace it are the vocabulary of the rest of the semester. TTFT, time to first token, is dominated by prefill and by queueing ahead of it. TPOT, time per output token — the synonym is inter-token latency, ITL — is the steady-state decode cadence; 22 ms is about 45 tokens/s, roughly a fast reader's pace. Throughput is aggregate tokens per second, which is what the operator pays for. Goodput counts only requests that met their SLO, and it is what catches a server posting excellent throughput while missing every latency target it promised.

The tension is structural, and §2.15's table already shows it: batch 1 gives 4.0 ms TPOT and 249 tok/s; B = 29 gives 22.6 ms and about 1,280 tok/s. Five times the throughput, five and a half times the per-request latency, one GPU. No setting is good at both, which is why the required paper reports a frontier rather than a number.

The specific failure this produces is a TPOT spike from prefill interference. Put one 4,096-token prefill in the same batch as 29 ongoing decodes: §2.14 priced that prefill at 64.0 TFLOP, and at an optimistic 50% of the H100's dense BF16 peak — an illustrative efficiency assumption, not a measurement — that is 64.0e12 ÷ 494.5e12 ≈ 129 ms. Added to the 22.6 ms decode step, the step takes about 152 ms, so all 29 sequences see a 6.7× TPOT spike because a stranger's prompt arrived. Two well-known answers exist — chunked prefill, which splits the prefill so no single step is dominated by it, and prefill/decode disaggregation, which runs the phases on separate pools — and both are Oct 14. Which means: report percentiles, never means. A mean TPOT of 25 ms is consistent with a smooth stream and with one that spikes to 150 ms every few seconds, and those are different products.

Now price it. Let R be the rented cost of one GPU per hour and G the number of GPUs:

Configuration	Throughput	$/Mtok	vs. best
Batch 1, MHA	249 tok/s	1.117 · R	21×
B = 29, MHA	1,280 tok/s	0.217 · R	4.0×
B = 116, GQA-8	5,130 tok/s	0.054 · R	1×

A 21× cost gap between a naive deployment and a well-configured one, on identical hardware and an identical model. Nothing in that table is a research result; it is §2.15's arithmetic divided by 3,600. Energy behaves the same way and necessarily so, since it is the same denominator: at a 700 W board TDP, batch 1 burns 700 · 0.00402 ≈ 2.81 J per token and the GQA-8 batch of 116 burns 0.14 J per token — the same 21×. Every optimization in Part II reduces FLOPs, bytes, or devices per unit of work, which is why Lecture 1 could claim that efficiency and sustainability are the same activity.

Finally, re-read Lecture 1's scaling arithmetic from the serving side, because the conclusion inverts. Chinchilla's D ≈ 20N is a statement about how to spend a training budget and nothing else. Lecture 1 showed training compute 6ND is overtaken by serving compute 2N per token at T = 3D generated tokens — N cancels — which under Chinchilla is T ≈ 60N. For the reference 7B that is 4.0 × 10¹¹ generated tokens: weeks for a popular service. So for any model with real usage, the serving term dominates the lifetime bill, and the design rule that follows is the opposite of the training-optimal one: a heavily served model should be smaller than compute-optimal and trained far past Chinchilla. Training a 7–8B model on trillions of tokens is indefensible as a training-budget decision and obvious as a serving decision. That is why the open-weight models you can actually run are the sizes they are.

Instructor notes

Minutes: 8, and it is the last thing they hear. Board: Four metric names. Then a timeline of 29 decode steps at 22.6 ms with one 129 ms prefill dropped in — the picture beats the arithmetic. Then the three-row cost table, and circle "21×". Ask the room: "Your server reports 5,000 tokens/s and a 25 ms mean TPOT. What has it not told you?" The tail, the batch composition, whether any request met its SLO. Expect confusion: Students read Chinchilla as a law about what models should be. Say: "It is a law about training budgets. Add a serving term and it tells you to overtrain a smaller model." Close on: "Twenty-one times, same hardware, same weights. That gap is the course."

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Reference 7B parameters	6,738,149,376 ≈ 6.74B	32 · 202,375,168 + 262,144,000
Reference 7B bf16 weights	13.5 GB	N · 2 bytes
MLP share of parameters	2/3 (135.3M of 202.4M per layer)	3·d_model·d_ff vs 4·d_model²
Forward FLOPs per token	≈ 2N = 13.5 GFLOP	one multiply + one add per weight
Training compute	C ≈ 6ND	2 forward + 4 backward
Attention/linear crossover	S = 24,704	202,375,168 ÷ (2 · 4096)
Persistent training state, Adam mixed precision	16 bytes/param (14 without the gradient)	2 + 2 + 4 + 4 + 4
Reference 7B persistent footprint	107.8 GB, against 80 GB of HBM	N · 16
Activations, reference 7B	≈ 3.69 MB per token	(6·d_model + 3·d_ff) · 2 · L
Ridge point, dense BF16	153 FLOP/byte (A100), 295 (H100)	peak FLOP/s ÷ bandwidth
Batch-1 decode floor	4.0 ms/token H100, 6.6 ms A100	13.5 GB ÷ bandwidth
Batch-1 decode utilization	1 FLOP/byte → 0.34% of H100 peak	1/ridge, independent of N
Prefill intensity at S = 4096	≈4,100 FLOP/byte (64.0 TFLOP ÷ 15.6 GB)	14× past the ridge
KV cache per token	512 KiB MHA, 128 KiB GQA-8	2·L·n_kv·d_head·b
KV budget and concurrency at 4K	62.5 GB → 29 MHA, 116 GQA-8	80 − 13.5 − 4
Batch needed vs. batch affordable	295 vs. 29	§2.14 against §2.15
Prefill interference spike	129 ms prefill on a 22.6 ms step → 6.7× TPOT	64.0 TFLOP at 50% peak
Naive-vs-tuned serving cost	21× (1.117·R → 0.054·R per Mtok)	§2.16
Energy per token	2.81 J (batch 1) → 0.14 J (GQA-8, B = 116)	700 W ÷ throughput
Training/serving break-even	T = 3D ≈ 60N → 4.04e11 tokens for the 7B	Lecture 1 §1.6

Self-check

Exercises

1. Grouped-query attention, recounted. Redo §2.4's parameter count with n_kv = 8 key/value heads instead of 32, keeping 32 query heads and d_head = 128. Report the new per-layer count, the new N, the bf16 weight bytes, and the percentage reduction. Then state precisely what does not change: the query head count, the attention score and value FLOPs of §2.5, and the crossover sequence length — and say what does change dramatically, with the number. Solution sketch: Q and the output projection stay d_model² = 16,777,216 each; K and V shrink to d_model · (n_kv · d_head) = 4096 · 1024 = 4,194,304 each. Attention per layer = 2·16,777,216 + 2·4,194,304 = 41,943,040, down from 67,108,864. Per layer 41,943,040 + 135,266,304 = 177,209,344; × 32 = 5,670,699,008; + 262,144,000 = N = 5,932,843,008 ≈ 5.93B, or 11.87 GB in bf16 — 12.0% fewer parameters (805,306,368 saved). Unchanged: all 32 query heads still attend over all S positions, so 4·d_model·S per layer per token is identical, and the crossover moves only because per-layer parameters fell — 177,209,344 ÷ 8192 = 21,632. What changes dramatically is the KV cache: 2·32·8·128·2 = 131,072 bytes per token, 128 KiB instead of 512 KiB, a 4× reduction, which is the entire reason the architecture exists and is the inference half's KV-cache accounting.
2. fp32 throughout. Redo §2.8's budget with no mixed precision: fp32 weights, fp32 gradients, fp32 Adam moments, fp32 activations. Report the persistent bytes per parameter, the total persistent footprint, activations at B = 1 / S = 4096, and the peak arithmetic throughput available. Then answer the question the exercise is really asking: what does mixed precision actually save? Solution sketch: Pure fp32 needs weights 4 + gradients 4 + first moment 4 + second moment 4 = 16 bytes per parameter, with no master copy required — identical to mixed precision's 16, so persistent state is 107.8 GB either way. Activations double: 57,600 · 4 · 32 = 7.37 MB per token, so 4,096 tokens cost 30.2 GB instead of 15.1 GB, and the total rises from 123.2 GB to about 138.5 GB. The real saving is elsewhere: arithmetic drops from 989 TFLOP/s dense BF16 to 495 TFLOP/s TF32 tensor (2.0×) or 67 TFLOP/s FP32 non-tensor (14.8×), and every activation-carrying byte of memory traffic doubles. Mixed precision is a throughput-and-activations optimization, not an optimizer-state optimization — a result most people guess backwards.
3. Is the step launch-bound? Using §2.9's conventions — 15 kernels per layer forward, 2× that backward, 32 layers, ~60 extra launches for embeddings and loss, a fused optimizer, and 5 µs per launch — compute total launches and launch time per step. Then compute the step's compute time at 40% of the H100's dense BF16 peak for B·S = 32,768 tokens and for B·S = 512 tokens, and give the launch-overhead fraction in each case. Finally, find the tokens-per-step at which the two are equal. Solution sketch: Launches = 32 · (15 + 30) + 60 = 1,500; at 5 µs that is 7.5 ms. Compute: C = 6 · 6.738e9 · 32,768 = 1.325e15 FLOP ÷ (0.4 · 989e12) = 3.35 s, so launches are 0.22%. At 512 tokens: C = 2.07e13 ÷ 3.956e14 = 52.3 ms, so launches are 14% — the same model, the same code, an order of magnitude worse. Equality at 6 · 6.738e9 · D = 7.5e-3 · 3.956e14 → D ≈ 75 tokens per step. An unfused per-tensor Adam adds roughly 2,900 launches and moves that threshold to about 220 tokens.
4. Move the crossover. Recompute §2.5's attention/linear crossover for d_ff = 4 · d_model = 16384, holding everything else fixed. Report the new per-layer parameter count, the new N, and the new crossover. Then explain why the crossover moved by almost exactly the same factor as N. Solution sketch: Per layer = 4·4096² + 3·4096·16384 = 67,108,864 + 201,326,592 = 268,435,456; × 32 = 8,589,934,592; + 262,144,000 = N = 8,852,078,592 ≈ 8.85B. Crossover = 268,435,456 ÷ (2 · 4096) = 32,768 tokens, up from 24,704 — a factor of 1.327, against N's factor of 1.314. They track because the crossover is per-layer parameters ÷ (2·d_model), and d_model is fixed, so the crossover is proportional to per-layer parameters; the tiny discrepancy is the fixed 262M of embeddings diluting N's growth. Fattening the MLP pushes the point where attention starts to matter further out, in exact proportion to the parameters you added.
5. From 6ND to break-even. Train the reference 7B at the Chinchilla ratio D ≈ 20N. Compute D, the training FLOPs, the H100-hours at 40% utilization, and the board-level energy at the 700 W limit. Then state the break-even output from Lecture 1's T = 3D and check it against the figure the inference half quotes. Solution sketch: D = 20 · 6.738e9 = 1.348e11 tokens (135B). C = 6 · 6.738e9 · 1.348e11 = 5.45e21 FLOP. At 0.4 · 989e12 = 3.956e14 FLOP/s: 5.45e21 ÷ 3.956e14 = 1.377e7 s = 3,826 H100-hours, which at an illustrative $2/GPU-hour — substitute your own rate — is order $7,700, and at 700 W is 3,826 · 0.7 = 2,678 kWh of board energy, boards only. Break-even is T = 3D = 4.04e11 generated tokens, about 404 billion, matching the inference half's ≈405 billion. The check that matters: serving those tokens costs 2 · 6.738e9 · 4.04e11 = 5.45e21 FLOP, equal to the training run, as T = 3D requires.
6. Size a deployment. Serve the reference 7B under GQA-8 on H100s at 30 requests/s, 2,000 prompt tokens and 300 output tokens each, with a TPOT SLO of 20 ms. Compute the decode batch the arrival rate implies, the batch the memory permits, the batch the SLO permits, which of the three binds, the number of GPUs, and the $/Mtok. Then state which fraction of GPU time goes to prefill, and what that implies about agent traffic. Solution sketch: Each request occupies a decode slot for 300 · TPOT; by Little's law the concurrency needed is 30 · 300 · 0.020 = 180 sequences. Memory: 2,300 tokens × 128 KiB = 0.294 GB each, so 62.5 ÷ 0.294 = 212 fit per card. SLO: the step must finish in 20 ms, and bytes per step are 13.5 GB + B · 0.294 GB, so 20 ms · 3,350 GB/s = 67.0 GB gives B ≤ (67.0 − 13.5) ÷ 0.294 = 182 — so the SLO binds, at B ≈ 182 against the 180 required, i.e. one card's decode capacity is just met. Prefill is the extra term: 30 req/s · 2,000 tokens · 13.48 GFLOP = 809 TFLOP/s of prefill demand against 494.5 TFLOP/s per card at 50% peak, so prefill alone needs 1.64 cards; decode needs 1.0; total ≈2.7, so 4 H100s with headroom for the interference of §2.16. At an illustrative R: aggregate 9,000 output tok/s over 4 cards → 4·R·1e6 ÷ (3600 · 9000) = 0.123·R per Mtok. Prefill is ~62% of the fleet's FLOPs while producing zero output tokens — and an agent, which re-sends a growing transcript every step (Lectures 3–5), is the limit case of that ratio. Doubling the prompt length is the worst input this deployment can receive.
7. Where the bytes go, and where they will not go. For the reference 7B on one H100 at S = 4096, compute the bytes moved per decode step at B = 1, 29 (MHA), and 116 (GQA-8); the resulting TPOT; the achieved arithmetic intensity of the whole step; and the weight share of the traffic. Then explain what the trend in the weight share says about which optimization matters at which batch size. Solution sketch: Bytes = 13.5 GB of weights + B · KV-per-sequence. B = 1 MHA: 13.5 + 2.15 = 15.7 GB → 4.7 ms; intensity 15.6 GFLOP ÷ 15.7 GB ≈ 1.0; weights 86%. B = 29 MHA: 13.5 + 62.4 = 75.9 GB → 22.7 ms; intensity 29·15.6 ÷ 75.9 ≈ 6.0; weights 18%. B = 116 GQA-8: 13.5 + 62.3 = 75.8 GB → 22.6 ms; intensity 116·15.6 ÷ 75.8 ≈ 24; weights 18%. The weight share collapses from 86% to 18% as the batch grows, so weight-only quantization is a low-batch optimization and cache reduction is a high-batch one — halving b for the weights at B = 116 buys about 9% of a step, while halving it for the cache buys about 41%. This is the arithmetic behind the split decision that Nov 9 asks you to make explicitly.

Reading guide

Efficiently Scaling Transformer Inference — required. The one required reading of the meeting, and the text for §2.11–§2.16. Read §2 and §3 and do not try to memorize the partitioning layouts; reconstruct the cost model instead — for each layout, what does it cost in compute, in memory traffic, and in collective communication, and which term dominates at which batch size? That is the reasoning the whole paper is an instance of, and it is the reasoning §2.16 asks you to do on one card. The multi-query attention discussion is §2.15's GQA argument in its original form. Skip the TPU mesh notation and the PaLM-specific results on a first pass. Hold this question: the paper reports a Pareto frontier rather than a best configuration — what is on the two axes, and which of their layouts would you pick for a chat product, and which for an offline batch job?

Attention Is All You Need — optional. Read §3, the model architecture, and nothing else on the first pass. You want the shapes: what Q, K, and V are, why the scores are divided by √d_head, and how multi-head attention concatenates. Figure 2 is the one to study — the left panel is §2.2's miniature drawn as a dataflow. Skip the machine translation results, the training details, and the positional-encoding discussion, all of which have been superseded. Hold this question: the paper's model is an encoder-decoder and ours is decoder-only, so which parts of §3 does a modern language model actually keep?

PyTorch and TensorFlow — optional, and read as a pair. Read TensorFlow's §2 and §3 for what a dataflow graph buys you — placement, fusion, and whole-program optimization — then read PyTorch's §2 and §3 for the argument that giving all of that up was the right call. In the PyTorch paper the sections that matter to us are the ones on how much performance an eager design can recover: the caching allocator and the multiprocessing and autograd internals. Skip both papers' benchmark tables; the hardware is a decade old and the numbers do not transfer. Hold this question across both: what was each system optimizing, and was it the resource that was actually scarce? That is the question How to Read a Paper calls the first pass, and it is what the paper discussion guide will ask presenters for all semester.

Looking ahead

Sixteen sections, and the last six are the ones Part II starts from. Sep 14 and Sep 16 turn to agents, whose token bills you can already price with 2N — and whose request streams turn out to be the most extreme version of §2.16's prefill-heavy traffic. Everything deferred here has a date. Sep 28 (LLM serving basics) takes §2.15's 29-versus-295 as its opening problem and shows what an allocator that pages the KV cache does to it. Sep 30 and Oct 5 are where §2.3's 15× byte ratio and §2.12's roofline become an actual kernel — the roofline paper is required reading on the second of those. Oct 7 and Oct 14 are batching and scheduling, where §2.9's launch counting reappears as CUDA graphs and §2.16's 6.7× interference spike gets its two answers. Oct 28 and Nov 9 shrink the cache and the weights, which is §2.10's question asked about a finished model rather than a training run. Nov 11 attacks the decode floor of §2.13 from the other side, by reading the weights fewer times per token. And parallelism and ZeRO, the answer to §2.8's 107.8 GB, are in optional content rather than in lecture.

Assignment 1 is in flight, due Sun Sep 20, 11:59pm (see assignments). One number to carry out of the room: twenty-nine against two hundred ninety-five. Every serving lecture for the next two months is an attempt to close that gap.