The first half of today's meeting ends on a promise the arithmetic could not keep by itself: halving the bytes per weight halves the decode floor, and nothing in that division says the model survives it — or that the kernel does. Today we close both halves of the 4-bit story that are usually told separately. The algorithm half: how GPTQ and AWQ reach 4 bits with no retraining, by two opposite philosophies — compensate every rounding error, or protect the weights that matter. And the systems half: QServe's finding that a 4-bit checkpoint can serve slower than an 8-bit one when the dequantization lands in the wrong place in the GEMM main loop. The thesis of the day: a quantization paper's accuracy table is half the story; the kernel's placement of dequant is the other half. By the end you should be able to run GPTQ's update rule on a toy layer by hand, explain why AWQ scales channels instead of mixing precisions, compute where dequantization cost lands on an H100, and decide from a workload's batch size and prefill/decode mix whether 4-bit pays at all.

Where this sits

Lecture 2 established the machinery this lecture spends: the 4.0 ms batch-1 decode floor for the reference 7B on an H100, the precision table showing int4 promising ~1.0 ms, the KV-cache batch ceiling of 29 (MHA) versus 116 (GQA-8), the 29-versus-295 gap between the batch memory allows and the batch the ridge demands, and the observation that at large batch the step is nearly all cache traffic. Today's first half introduces quantization formats, round-to-nearest, and the quality question. The rest finishes the job: the two canonical post-training algorithms that make 4-bit weights respectable, the kernel accounting that decides whether the promised milliseconds arrive, and an honest verdict on pruning. Speculative decoding (Nov 11) attacks the same floor from the other side — fewer passes per token rather than fewer bytes per pass — and the two multiply.

Instructor notes — Timing plan

75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221). Instructor-led: the instructor walks GPTQ and QServe, keeps the board honest, and holds the clock.

Time	Segment	Notes
0–8	Framing + §15.1	Thesis sentence on the board first. Recap Lecture 2's floor table; derive 4.125 effective bits and the 3.47 GB / 1.04 ms honest floor before any paper is named.
8–25	§15.2 GPTQ deep-dive	Walk the algorithm. Run the two-weight toy live — it is exact arithmetic. H = 2XXᵀ and its three assumptions must end up on the board.
25–38	§15.3 AWQ + SmoothQuant cameo	The philosophy table (compensate vs protect). Why scaling beats mixed precision is the discussion hinge.
38–60	§15.5–15.6 QServe	Centerpiece — never cut. Board model of dequant cost (0.8 ms vs the 1.04 ms floor), then progressive quantization as the answer.
60–70	§15.8 + §15.9 synthesis	Pruning verdict in five minutes; the 2×2 decision rule; only the headline rows of §15.7 (135/540, B ≈ 148).
70–75	Discussion seeds	Pick two seeds live; assign the rest with the reading.

Reading-only, not scheduled: §15.4 (rotations/SpinQuant) and the full table and cost rows of §15.7 — the class gets the 135/540 headline and the B ≈ 148 ridge line only.

If running long: compress §15.2's toy to its punchline ("correlation is the budget compensation spends"). Protect §15.5–15.6 — the dequant accounting must be on the board when the bell goes.

Learning objectives

By the end of this class you should be able to:

1. Run GPTQ's quantize-then-compensate update on a small layer by hand, state where H = 2XXᵀ comes from, and name the three assumptions it encodes about calibration data.
2. Explain why AWQ protects salient channels by per-channel scaling rather than mixed precision, and compute what bounds the scale.
3. Compute, from an instruction-count model, where dequantization cost lands in a W4A8 GEMM on an H100 — and predict the batch regime where a naive 4-bit kernel loses to an 8-bit one.
4. Recompute the serving stack of Lecture 2 — floor, batch ceiling, TPOT, cost per Mtok — for the reference 7B at W4A8KV4.
5. Decide, from a workload's batch size and prefill/decode mix, which of W4, A8, and KV4 pays, and in what order.
6. Argue with bytes why 2:4 structured sparsity, despite living in the hardware, lost to int4 for LLM serving.

15.1 Where round-to-nearest dies, and the shape of the fix

Start from what Lecture 2 already sold us. The reference 7B holds 13.5 GB of bf16 weights; the batch-1 decode floor on an H100 is 13.5 ÷ 3,350 = 4.0 ms per token, and the precision table promised that int4 — 3.37 GB — brings it to ~1.0 ms and 994 tok/s, while freeing 10 GB of HBM for cache. The prize is just under 4× on the floor and more than 4× on batch capacity once the cache is quantized too (§15.7).

First, make the int4 number honest, because raw 4-bit values cannot stand alone: every group of weights shares a scale that maps the 16 integer levels back to real values. Take the standard layout — group size g = 128, one fp16 scale per group:

Second, the quality problem. Sixteen levels per group is not many, and the levels are placed by the group's extremes. Naive round-to-nearest (RTN) — quantize each weight independently to its nearest level — is serviceable at 8 bits and collapses at 4, and one line of arithmetic shows why:

Grouping already contains weight outliers somewhat — poisoning 128 neighbors instead of a whole channel is what g = 128 buys, and why we pay the 3% scale tax. The harder problem is that activation outliers are per-channel and systematic: a handful of input channels carry values orders of magnitude larger than the rest, on every token, so the weights multiplying them matter far more than their own magnitudes suggest — and no weight-side grouping can see that. That is LLM.int8()'s emergent-feature finding: the outlier dimensions are few, consistent across layers, and switch on as models pass a few billion parameters, which is why naive int8 degrades on large models and not on small ones. Its answer was to run those dimensions in fp16 and everything else in int8 — where §15.3 picks the argument up. Both of today's algorithm papers are answers to one question: who absorbs the outliers?

	GPTQ	AWQ
Philosophy	Error compensation — fix the damage after each rounding, using curvature	Importance protection — prevent damage to the ~1% of weights that matter, using activations
Information used	Calibration activations as X·Xᵀ (second moments, correlations)	Calibration activations as mean magnitudes per channel
Output format	Uniform int4, grouped scales	Uniform int4, grouped scales
Retraining	None	None

Both are post-training, both see only a few hundred calibration sequences, and both leave a format the kernel can stream. Hold onto that last row: it is the bridge to the second half of the lecture.

No retraining means the only information either method has is what those calibration sequences reveal. Everything both papers do is a bet on what a few hundred sequences represent — keep that in view when the accuracy tables appear.

Instructor notes

Minutes: within the 0–8 framing block. Board: the thesis sentence, then 3.47 GB → 1.04 ms → 4.125 bits derived live. Leave the two-column philosophy table up all class. Ask the room: "You have 16 levels and one weight 8× the rest. Where do the levels go?" The answer motivates everything that follows.

15.2 GPTQ: quantize a column, bill the survivors

The lineage is three sentences long. Optimal Brain Surgeon (Hassibi and Stork, 1993) pruned one weight at a time and used the inverse Hessian of the loss to update every remaining weight, compensating the removal. Optimal Brain Quantization (2022) replaced "prune" with "quantize", greedily picking the weight whose rounding hurt least. GPTQ made that tractable at 175B scale with two moves: (a) quantize columns in a fixed left-to-right order instead of greedily — the order affects accuracy surprisingly little, and since the Hessian depends only on the layer's inputs, a fixed order means every output row of the weight matrix shares the same Hessian and all rows update in parallel; (b) batch the compensation updates lazily in blocks and precompute the needed inverse-Hessian information via a Cholesky factorization, which is what keeps the accumulated updates numerically sane at scale. The result, per the paper: a 175B-parameter model quantized in about four GPU-hours on a single A100 — no gradients, no fine-tuning.

Be precise about what "second-order information" means here, because it is less than it sounds. GPTQ's objective is layer-wise, not end-to-end: for each linear layer with weights W and calibration inputs X, minimize the reconstruction error ‖WX − ŴX‖² subject to Ŵ living on the quantization grid. The Hessian of that objective with respect to one row of W is

Second-order in the layer's reconstruction loss, not in the language-modeling loss. That substitution buys tractability and costs three assumptions, each attackable:

1. Layer errors are independent. The objective treats each layer alone; in reality each quantized layer feeds its errors into the next layer's X, and they compound. (GPTQ mitigates this by quantizing layers sequentially against the already-quantized network's activations, but the objective still never sees the end-to-end loss.)
2. Calibration covers deployment. H is built from a few hundred calibration sequences; a domain those sequences never visited contributes nothing to X·Xᵀ and gets no compensation — or worse, gets compensation tuned for someone else's correlations.
3. The loss is locally quadratic. That is what makes a Hessian the right object at all; large rounding steps on a coarse grid stretch the approximation.

The algorithm, one column at a time within each block: quantize weight w_q to the grid; compute the scaled error; charge it to the not-yet-quantized weights in the same row.

The update says: weights whose inputs are correlated with w_q's input can absorb its rounding error. Correlation is the budget that compensation spends — which the following toy makes exact.

A two-weight toy you can check by hand

One output neuron, two weights w = (0.6, 0.6), quantization grid = the integers {0, 1, 2}, and calibration data in which the two inputs are perfectly correlated: every sample has x₁ = x₂. Perfect correlation makes the optimal compensation coefficient exactly 1 — a unit of error on w₁ is fully absorbable by w₂. (It also makes H = 2XXᵀ exactly singular, which is why real GPTQ adds dampening — about 1% of the mean diagonal added to H — before inverting; the coefficient-1 update is the well-behaved limit.)

The failure mode, for free

Now break the toy: if x₁ and x₂ are uncorrelated, the off-diagonal of H is zero, the compensation coefficient is zero, w₂ never moves, and GPTQ reduces exactly to RTN on this layer. The second-order machinery is not magic — it is a mechanism for moving error onto correlated coordinates, and it buys nothing where correlations do not exist. Transformer activations happen to be richly correlated, which is why the mechanism earns its keep in practice: at 4 bits the largest models land within a few tenths of a perplexity point of fp16, with the gap growing as models shrink and as bits drop toward 3.

Punchline: GPTQ is a proof that 4-bit weights need no gradients — only a Cholesky, a few hundred calibration sequences, and an afternoon.

Instructor notes

Minutes: 17 (8–25). Walk the algorithm; write the toy on the board. Board: the update rule, then the toy in two columns (RTN | GPTQ). Circle "0.8 → 0.2" and write "bought with correlation" under it. Ask the room: "What happens to this toy if the calibration set has x₁ = x₂ but deployment traffic has x₁ = −x₂?" Compensation now adds error — assumption (ii) made vivid. Expect confusion: "Second-order" gets read as second-order in the LM loss. Say: "It is the Hessian of a per-layer least-squares problem. The LM loss never appears."

15.3 AWQ: not all weights are equal, and the fix is a scale, not a format

AWQ starts from an observation rather than an objective. Keep a tiny fraction of weight channels — the paper's figure is 0.1–1% — in fp16 and quantize the rest to 4-bit RTN, and quality largely recovers. The catch is which fraction: selecting by activation magnitude — the channels whose inputs are systematically large — works; selecting the same fraction by weight magnitude does not. The salient weights are salient because of what multiplies them, not because of their own size. This is the activation-outlier problem of §15.1 restated as a discovery procedure.

But "keep 1% in fp16" is the right diagnosis and the wrong prescription. Mixed precision means fp16 islands scattered through an int4 tensor: the memory layout fragments, the kernel needs two paths, and the hardware efficiency that motivated 4-bit in the first place leaks away. AWQ's move is to get the protection without the format change: per-channel scaling. Multiply a salient weight channel by s > 1 before quantization, and fold the inverse 1/s into the operation that produces that channel's activation (the previous layer's output scale — algebraically free). The function computed is unchanged; the channel's representation on the grid is not.

The one-line error model says why this works and what bounds it. Per-group RTN error for a weight's output contribution is proportional to the grid step times the multiplying activation:

The Δ′ ≈ Δ proviso — the scaled weight must not become the new group maximum — is the whole design tension: too small an s protects nothing, too large an s stretches the grid and taxes the other 127 weights in the group (self-check 3 makes this quantitative). So AWQ does not derive s; it searches for it on calibration data — a single exponent α ∈ [0, 1] applied to each channel's mean activation magnitude, grid-searched to minimize the layer's output MSE. No gradients, no regression, one knob per layer.

The philosophy contrast with GPTQ is worth running as discussion rather than reading. GPTQ reacts to error using X·Xᵀ — a rich statistic (full second moments) that needs more calibration data and can overfit it; AWQ's stated critique of reconstruction-based methods is exactly this. AWQ prevents error using only mean activation magnitudes per channel — a much weaker statistic, and more robust across distribution shift for the same reason. Neither touches the LM loss; both bet on calibration data, but they bet different amounts on different statistics.

One cameo prepares §15.6. SmoothQuant (optional reading) is the same scaling algebra run in the opposite direction: it migrates activation outliers into the weights — per-channel s_j balancing max|X_j| against max|W_j| with a balance exponent α ≈ 0.5 — so that both sides become 8-bit-quantizable and W8A8 works on integer tensor cores. AWQ migrates quantization difficulty away from salient weights for W4; SmoothQuant migrates it away from activations for A8. Same lever, two owners of the outlier problem — and QServe, needing W4 and A8 and KV4, will pull it in both directions at once. On speed, AWQ's own serving stack (TinyChat) reports roughly 3×-class gains over fp16 baselines on memory-bound decode — consistent with the bytes ratio, which is the only place such a gain can come from.

Punchline: AWQ wins not by a better error model but by refusing any format the kernel cannot stream.

Instructor notes

Minutes: 13 (25–38). Board: err ∝ Δ·|x|/s, then "until s·w is the new max" next to it. Add GPTQ-reacts / AWQ-prevents to the standing table. Ask the room: "Both methods use calibration data. Which survives a language it never saw?" Steer to: AWQ's mean-magnitude statistic is coarser and therefore harder to overfit — robustness by poverty. Expect confusion: students conflate AWQ's weight-side scaling with SmoothQuant's activation-side scaling. The direction of migration is the difference; say it twice.

15.4 Rotations: spreading outliers before you quantize (reading-only)

There is a third verb, and it is one screen's worth here because it is optional reading and Blackwell-era relevant. Computational invariance: insert a rotation R and its inverse around a linear operation and the network's function is unchanged — (W R)(Rᵀ x) = W x. But the distributions being quantized change completely: a rotation mixes a few outlier channels into all channels, flattening the distribution so a uniform grid finally fits it. Random or Hadamard rotations already help substantially (the QuaRot lineage); SpinQuant's contribution is that random-rotation quality has high variance, so it learns the rotations — optimizing on the manifold of orthogonal matrices (Cayley-style parametrization) with the network function provably fixed.

Why this matters for the systems half: rotations are the main published route to W4A4 — quantizing activations to 4 bits is the outlier problem at its worst, and neither compensation (GPTQ) nor scaling (AWQ/SmoothQuant) has anywhere left to hide an outlier when both sides of the multiply are down to 16 levels. SpinQuant-class methods close a large fraction of the W4A4 quality gap that scaling methods leave — at the price of extra rotation matmuls at inference (some foldable into existing weights, some not) and an optimization step that plain GPTQ/AWQ do not need.

Punchline: GPTQ compensates outlier damage, AWQ shelters outliers, rotations abolish them — three verbs, one enemy.

Instructor notes

Minutes: 0 — reading-only, not scheduled. Board: if a question drags it in, write (WR)(Rᵀx) = Wx and "same function, different distributions" — that is the entire section in one line. Ask the room: nothing; redirect W4A4 questions to discussion seed 2, which is where the class time for this lives.

15.5 The kernel trap: dequant lands in the main loop

Now the half of the story the accuracy tables do not show. Int4 weights cannot feed tensor cores directly — no mainstream matrix unit multiplies int4 weights by int8 or fp16 activations without the weights first being expanded. So a W4A8 GEMM must dequantize every weight INT4 → INT8 inside the GEMM main loop, on every pass over the weights — and on Ampere and Hopper that dequantization executes on CUDA cores, the scalar/vector slow path, while the tensor cores do the MACs. Weight bytes shrank 2× versus W8A8; per-weight work appeared where the bytes arithmetic has no column for it.

Put numbers on it with an explicitly labeled model — an illustrative instruction count, not a measurement. Assume a straightforward dequant costs ~4 CUDA-core instructions per weight: unpack/mask the nibble, subtract the zero-point, multiply by the scale, convert to the accumulation type.

Read that against the promise. Unoverlapped, the "4× faster" int4 checkpoint runs at 1.04 + 0.8 ≈ 1.84 ms — a 2.2× win instead of 3.9×, nearly half the gain surrendered to instruction issue. Even a well-overlapped kernel does not escape cleanly: those 27 billion instructions contend for the same issue slots the memory pipeline needs for address generation and predication, so the floor is approached, not reached.

Large batch is worse, not better, because the dequant cost is per-pass-over-the-weights — it does not amortize with B, while the GEMM's useful work is now on the clock. Run the same model at the two batch extremes and one phase:

The middle row is why naive W4A8 kernels can lose to plain W8A8 at exactly the batch sizes where serving operates — the observation QServe's profiling opens with, finding dequantization a first-order term in prior W4A8 main loops. The 4-bit checkpoint had better accuracy-per-byte; the 8-bit one had a main loop with nothing in it but MACs. And the bottom row is how the trap stays hidden: a paper that benchmarks prefill-heavy workloads, or reports only perplexity, never meets it. The trap lives in decode — exactly where 4-bit's motivation lives.

Punchline: bytes tell you the floor; the instruction stream tells you whether you may stand on it.

Instructor notes

Minutes: first half of the 38–60 centerpiece block. Board: the three-line model, boxed, with "ILLUSTRATIVE — 4 instr/weight assumed" written above it. Then 0.8 vs 1.04 side by side, then 0.8 vs 1.75 at B = 256. Ask the room: "The dequant cost is per weight per pass. What single number in this model would you measure first on a real kernel?" Instructions-per-weight — exercise 2 and seed 6. Expect confusion: "Why not dequantize once and cache the int8 weights?" — the tip above answers it; make whoever asks say where the cached copy would live, and the trap closes itself.

15.6 QServe: W4A8KV4, and paying for dequant once

QServe names its design point in the title: W4A8KV4 — 4-bit weights, 8-bit activations, 4-bit KV cache — and each term is an argument:

Term	What it buys	What it dodges
W4	Batch-1 bytes (the 1.04 ms floor) and 10 GB of freed capacity	—
A8	Int8 tensor peak, 1,979e12 ops/s on H100 — 2× bf16, for prefill and large-batch decode	A4's outlier cliff: rotations required, accuracy still bleeding
KV4	The batch ceiling — at serving batch the step is mostly cache traffic (§15.7)	—

Not W4A4 — too much quality risk for an extra 2× on a resource (activation bytes) that rarely binds. Not W8A8 — leaves the weight bytes, the entire batch-1 and capacity story, on the table.

The answer to §15.5 is progressive (two-level) quantization of the weights. Level 1: quantize per-channel to the int8 range with fp16 scales — this is the envelope. Level 2: within it, quantize per-group (g = 128) from int8 down to int4 with int8 scales. Now the main-loop dequant is int4 → int8 entirely in cheap integer arithmetic — a subtraction and an int8 multiply per weight, with the ranges engineered ("protective range" clamping) so intermediates provably cannot overflow, which is what lets the kernel skip guard instructions. Compare against the naive path, int4 → fp16 with floating-point CUDA-core multiplies and a type conversion: fewer instructions, cheaper instructions, and register-friendly — the paper's "subtraction after multiplication" reordering exists purely to shave main-loop instructions further. Dequant still happens every pass; it just stops being expensive enough to matter.

The supporting cast, one line each, because each fixes a specific leak:

• SmoothAttention — SmoothQuant's migration applied to attention: K's outlier channels are scaled down and the compensation folded into Q (free, since Q·Kᵀ is bilinear), so 4-bit KV quantization stops destroying attention scores. KV4 is what makes the §15.7 capacity table real.
• Compute-aware weight reordering — store weights in exactly the order the GEMM's tiles consume them, so main-loop loads are wide and contiguous and the pointer arithmetic leaves the loop. Instruction-count discipline applied to addressing, not just dequant.
• Fused attention kernels that operate on int4 KV directly — the cache is dequantized in-register inside the attention kernel, never materialized at higher precision in HBM.

Check the storage while we are here, because the two-level scheme also wins the scale tax:

The measured result, stated at the fidelity we can defend: QServe reports large end-to-end serving-throughput gains over strong TensorRT-LLM-class baselines — tens of percent on 8B-class models rising to 2–3× on 70B-class — with the largest wins on the cheaper L40S part, whose headline is a cheap card matching or beating an A100 on dollars per token. The shape matters more than the digits, and it is worth reading twice: the gains grow with model size — exactly where weight traffic and dequant pressure grow — and they grow on the bandwidth-poorer card, which is the signature of a kernel fix rather than an accuracy fix. An algorithm-side improvement would have shown up uniformly; a main-loop fix shows up where the main loop hurt.

Punchline: QServe's contribution is an accounting identity — it moved dequantization off the loop's critical resource, and only then did the accuracy table's promise become a throughput.

Instructor notes

Minutes: second half of 38–60. Never cut. Board: two-level diagram — fp16 scale → int8 envelope → int8 scale → int4 — with "main loop crosses only this arrow" pointing at int4→int8. Ask the room: seed 3 — "If Blackwell dequantized int4 inside the tensor core, which parts of QServe survive?" (SmoothAttention, KV4 fused attention, and reordering survive; progressive quantization becomes hardware.) Expect confusion: W4A8 gets read as "activations quantized to help bandwidth." No — A8 is a compute decision (int8 tensor peak); the bandwidth story at low batch is almost all W4, and at high batch almost all KV4.

15.7 What W4A8KV4 buys on the reference 7B (reading-only except the headlines)

Rerun Lecture 2's serving stack at 4-bit weights and 4-bit KV: one H100, S = 4,096, 4 GB reserved, weights 3.5 GB (§15.6's layout, rounded).

Quantity	bf16 (Lecture 2)	W4A8KV4	Ratio
Weights resident	13.5 GB	3.5 GB	3.9×
Free for KV	62.5 GB	72.5 GB	1.16×
Max B, MHA, 4K	29	72.5 ÷ 0.537 ≈ 135	4.7×
Max B, GQA-8, 4K	116	72.5 ÷ 0.134 ≈ 540	4.7×
Bytes/step at max B (GQA-8)	75.8 GB	3.5 + 540 · 0.134 = 75.9 GB	1.0×
TPOT at max B	22.6 ms	22.7 ms	1.0×
Aggregate throughput	5,130 tok/s	540 ÷ 0.0227 ≈ 23,800 tok/s	4.6×
Cost per Mtok	0.054 · R	1e6 · R ÷ (3600 · 23,800) = 0.0117 · R	4.6×

Sanity-check the compute so the table is legitimate: 540 × 13.48 GFLOP = 7.3 TFLOP per step ÷ 1,979e12 int8 ops/s ≈ 3.7 ms ≪ 22.7 ms — the step is still memory-bound, so dividing bytes by bandwidth was the right model. Against Lecture 2's naive batch-1 bf16 deployment at 1.117 · R, the well-configured quantized deployment is now ~95× cheaper per token — the 21× gap of Lecture 2, stretched by quantizing both terms of the traffic.

Two structural observations survive the 4× rescaling untouched. The knee — the batch at which KV traffic first equals weight traffic — sits at B = 3.5 ÷ 0.134 ≈ 26, essentially where bf16 had it (13.5 ÷ 0.537 ≈ 25): quantizing both terms by 4× moves the ceiling, not the shape. And at B = 540 the step is 95% KV traffic — Lecture 2's "96% cache" line, reborn one precision level down. KV4 is worth more than W4 at scale, and it is not close.

One row is genuinely new. Weight-GEMM decode intensity at W4A8 is B · 2N int8 ops ÷ N · 0.5 bytes = 4B ops/byte; the int8 ridge on H100 is 1,979 ÷ 3.35 ≈ 591, so the weight GEMMs cross into compute-bound at B ≈ 148 — and the memory permits 540. For the first time in this course, the batch the memory allows exceeds the batch the ridge demands. The step as a whole stays KV-bound (the 3.7-vs-22.7 check above), but the weight matrices — the term all of this machinery was invented for — finally saturate the tensor cores. Quantization is what closes Lecture 2's 29-versus-295 gap on paper; §15.5's dequant tax is what decides it in silicon.

Punchline: the ceiling moved 4.7×, the knee did not move at all, and past B ≈ 26 the only bytes left worth attacking are the cache's.

Instructor notes

Minutes: headlines only, inside the 60–70 block — the table and cost rows are reading. Board: three numbers: "135 / 540" (was 29/116), and "B ≈ 148 < 540" with the caption "the ridge is finally reachable." Ask the room: "The knee didn't move. Why not?" Both traffic terms shrank 4× — the ratio is precision-invariant. Ten seconds, worth it. Expect confusion: "compute-bound at B ≈ 148" gets read as "the step is compute-bound at B = 540." No — the weight GEMMs are; the step is 95% KV bytes and remains memory-bound. Keep the two statements separate on the board.

15.8 Pruning: the road not taken

Pruning shares the goal — fewer bytes, fewer ops — and even has something quantization lacks: dedicated hardware. Ampere and Hopper tensor cores support 2:4 structured sparsity — two of every four weights zero — and the sparse spec-sheet numbers are exactly double the dense ones. This course has refused those numbers all semester; here is the honest accounting of why.

Do the bytes first. A 2:4-sparse layout stores, per 4 elements, the 2 surviving bf16 values (4 B) plus 2-bit position indices (0.5 B): 4.5 B versus 8 B dense — 56% of the bytes, so at best a 1.78× decode-floor gain (7.58 GB → 2.26 ms on H100). Dense int4 gives 3.9× (1.04 ms). So the ceiling on pruning's bandwidth win is less than half of quantization's — before quality enters.

Then quality. The one-shot pruning literature has real results — SparseGPT is GPTQ's own machinery (layer-wise reconstruction, Hessian-guided updates) aimed at zeroing weights instead of rounding them, and Wanda gets close with a far simpler weight-times-activation-magnitude criterion; both reach 50–60% unstructured sparsity with modest loss on the largest models. But 2:4 is a harsher, structured constraint — every group of four must sacrifice two — and without retraining it measurably costs quality on LLMs where int4 quantization costs almost none. Quality-matched, end-to-end serving wins for 2:4 over a quantized dense baseline are rare in the literature; the burden of proof sits with the sparse side and has largely gone unmet.

The comparison that settles it: int4 buys 4× fewer bytes at near-zero quality cost with mature kernels; 2:4 buys 1.78× fewer bytes at nonzero quality cost with ecosystem friction. Same engineering budget, strictly dominated for LLM serving. Pruning survives where quantization is already spent — sparse-plus-int4 stacking (exercise 5) — and in domains with retraining budgets and different accuracy economics.

Punchline: 2:4 exists in the hardware and mostly not in production LLM serving — a rare case of a shipped hardware feature losing to arithmetic.

Instructor notes

Minutes: ~5 inside the 60–70 block. Board: "4.5/8 = 56% → ≤1.78×" next to "0.5/2 = 25% → 3.9×". Two fractions end the debate. Ask the room: seed 5 — what result would revive 2:4? A quality-matched end-to-end win over dense-int4, which is the comparison the spec sheet hopes you will not run.

15.9 When 4-bit pays, and the discussion

The decision rule, as the 2×2 the whole lecture has been building:

Workload	W4	A8	KV4
Decode-heavy, low batch (bandwidth-bound)	Full win — floor 4.0 → ~1.0 ms, 4× capacity	Little — tensor peak is not the binding resource	Wins when contexts are long
Decode-heavy, large batch	Small per-step (past the B ≈ 26 knee), still real for capacity	Real once near the ridge — 2× int8 peak	The lever — the step is ~95% KV bytes
Prefill-heavy (agent traffic)	Pure tax — bytes do not bind, dequant is 0.6–1.2% overhead: harmless and pointless	The only win — 2× compute on the phase that dominates	Session capacity — Lecture 5's 13.4 GB stalled session becomes ~3.4 GB

Agent traffic deserves the flag: Lecture 5 measured prefill at ≈60× decode compute for an agent task, and Lecture 2's sizing exercise found mean prompt length the most sensitive input. For that workload, quantizing weights is a checkbox; A8 and KV4 are the money. Check which row your Assignment 4 system lives in before quantizing anything — that is seed 4.

Discussion seeds

1. Philosophy. GPTQ's compensation needs X·Xᵀ; AWQ's protection needs only E|x| per channel. Construct a deployment where each one's calibration assumption breaks — a domain shift, a language absent from the calibration set — and predict which degrades more gracefully, and why.
2. Honest W4A4. What would it actually require? Rotations for activation outliers (§15.4), a per-token activation quantizer on the critical path, and int4×int4 tensor-core support. Price each: which is algorithm work, which is kernel work, which is a hardware generation?
3. Algorithm or kernel? If Blackwell-class hardware dequantized int4 inside the tensor core, which parts of QServe survive? Which parts of GPTQ and AWQ become more valuable?
4. Whose accuracy table? Perplexity, zero-shot suites, or your own agent's task success rate — which do you trust to certify a 4-bit model for your Assignment 4 system, and what would it cost to measure?
5. Why did 2:4 lose? Could a 4:8 or 8:16 pattern with a retraining budget have won? What would the bytes arithmetic of §15.8 need to show before the answer changes?
6. Kill the model. §15.5 assumed 4 instructions per weight and a 33.5e12 instr/s issue ceiling. What measurement on a real kernel confirms or kills it? (Nsight Compute: issue-slot utilization, instruction mix in the main loop — and exercise 2 is the pencil version.)

Instructor notes

Minutes: 60–75 across §15.8, the 2×2, and the seeds. Board: the 2×2 table, drawn empty, filled by the room — call on people per cell rather than lecturing it. Ask the room: run seeds 1 and 3 live if the QServe discussion has not already consumed them; seed 4 is the exit question ("check your Assignment 5 system's row before it is due Dec 2"). Expect confusion: "quantize everything" as a default. The prefill-heavy row exists to break that reflex — W4's dequant is harmless there and also pointless, and the engineering time belongs to KV4 and A8.

Key takeaways

Numbers worth memorizing

Quantity	Value	Source
Effective bits, int4 at g = 128, fp16 scales	4.125 bits → 3.47 GB, 1.04 ms floor (H100)	4 + 16/128; §15.1
GPTQ toy: RTN vs compensated error	0.8 → 0.2 (4×), bought from correlation	§15.2
Dequant board model (illustrative)	4 instr/wt → ~0.8 ms/step on H100	27e9 ÷ 33.5e12; §15.5
Dequant share: batch-1 / GEMM at B = 256 / prefill	77% of floor / 46% / 0.6%	§15.5, exercise 4
W4A8KV4 ceilings, 7B, 1× H100, 4K	B = 135 MHA, 540 GQA-8; 23,800 tok/s; 0.0117 · R	§15.7
int8 ridge (H100) and W4A8 crossover batch	591 ops/byte → B ≈ 148	1,979 ÷ 3.35; 4B = 591
Weight-vs-KV knee	B ≈ 26 (was ≈ 25 at bf16)	3.5 ÷ 0.134
Two-level scale storage (QServe layout)	53 MB int8 + negligible fp16 → 3.42 GB total	§15.6
GPTQ cost at 175B scale	~4 GPU-hours, one A100, no gradients	§15.2, per the paper
2:4 sparsity byte ratio	4.5/8 = 56% → ≤ 1.78× floor gain	§15.8

Self-check

Exercises

1. Group-size tax. For g = 32, 128, 256 with one fp16 scale plus one int4 zero-point per group (20 bits/group), compute effective bits/weight, weight bytes for the reference 7B, and the H100 batch-1 floor. Solution sketch: bits = 4 + 20/g → 4.625 / 4.156 / 4.078; bytes = N · bits/8 → 3.90 / 3.50 / 3.44 GB; floors 1.16 / 1.05 / 1.03 ms. Shrinking g from 256 to 32 costs ~13% of the floor and buys finer grids (each outlier poisons 32 neighbors instead of 256) — the quality-versus-bytes knob, made explicit.
2. Kill the board model. Using §15.5's issue ceiling, find the instructions-per-weight k at which unoverlapped dequant time equals the 1.04 ms int4 batch-1 floor on H100. Then evaluate a QServe-style dequant at k = 2. Solution sketch: k · 6.74e9 ÷ 33.5e12 = 1.04e-3 → k ≈ 5.2 — a five-instruction dequant doubles batch-1 latency if unoverlapped. At k = 2: 0.40 ms = 39% of the floor, so even the cheap path must overlap with the weight stream to vanish; the requirement is that issue slots for dequant hide under memory latency, which is a schedulability property of the kernel, not of the format.
3. Long-context rescue. S = 32,768, MHA, one H100. How many sequences fit at bf16 weights + KV16, versus int4 weights + KV4? TPOT at the quantized ceiling? Solution sketch: KV16: 512 KiB × 32,768 = 17.2 GB/seq; 62.5 ÷ 17.2 → 3 sequences. KV4 + W4: 4.29 GB/seq; 72.5 ÷ 4.29 → 16 sequences. At B = 16: bytes/step = 3.5 + 68.7 = 72.2 GB → 21.6 ms TPOT → ~742 tok/s. KV4 turns "3 users" into "16 users" at an unchanged cadence — capacity, not speed, is what long context buys from quantization.
4. Prefill indifference. A 4,096-token prefill on the 7B: bf16 at 50% of 989 TFLOP/s versus W4A8 at 50% of the 1,979 int8 peak; then place the 0.8 ms dequant tax against both the prefill and the 1.04 ms decode step. Solution sketch: bf16: 64.0 TFLOP ÷ 494.5 = 129 ms; W4A8 ideal: 64.0 ÷ 989.5 = 65 ms (A8's 2× is the entire win). Dequant: 0.8 ÷ 65 ≈ 1.2% of prefill — invisible; 0.8 ÷ 1.04 ≈ 77% of the decode step — fatal. The same kernel tax is noise in one phase and the whole story in the other, which is why prefill-heavy benchmarks never find it.
5. The 2:4 counterfactual. Compute the H100 batch-1 floor for the 7B at 2:4-sparse bf16, dense int4, and a 2:4-sparse int4 stack (apply the same 4.5/8 byte ratio to the 3.47 GB layout). What evidence would the stack need before you would ship it? Solution sketch: 2:4 bf16: 0.5625 × 13.48 = 7.58 GB → 2.26 ms — loses to dense int4's 1.04 ms outright. Stack: 0.5625 × 3.47 ≈ 1.95 GB → 0.58 ms, a real further 1.8× if quality holds. The missing evidence: quality-matched, end-to-end evaluations against the dense-int4 baseline — the comparison the sparse literature mostly does not run (seed 5), and the burden is on the stack because it inherits both methods' quality costs.

Reading guide

Required — LLM.int8(). The first-half text, and the one paper today that is diagnosis rather than prescription. Take two things: the emergent-outlier measurement — a few hidden dimensions carrying magnitudes far above the rest, consistent across layers, appearing once models pass the few-billion-parameter mark, which localizes the int8 failure to large models specifically — and the two-part fix, vector-wise scaling plus a 16-bit decomposition for the outlier dimensions. Skim the emergence sweeps and the zero-shot tables for shape only. Hold: the fix ships two matmuls and a split layout; which of today's papers refuses to pay that, and what does it substitute?

Required — GPTQ. Read §1 for the OBS→OBQ lineage — the paper is honest that it is an engineering of prior ideas to a new scale. In the algorithm sections, extract three things and stop: the argument that a fixed column order sacrifices little accuracy while making H = 2XXᵀ shared across all rows; what lazy blocked updates plus the Cholesky reformulation buy in numerical stability and memory traffic; and the dampening trick for near-singular H. Skim the results tables for shape only — the quality gap shrinks with model size and blows up below 4 bits — and skip the OBC recap details. Hold: what is the calibration set silently assumed to represent, and which of your deployments violates it?

Optional — AWQ. Read the salient-channel study (activation- versus weight-magnitude selection — the paper's Table comparing the two is the finding) and the scaling derivation with its error model; skim the α-search; skip TinyChat internals. Hold: why is mixed precision the wrong implementation of the right observation?

Required — QServe. The profiling story is the centerpiece: read until you can redraw, from memory, where dequantization sits in a W4A8 GEMM main loop and which unit executes it. Then read progressive group quantization (two-level scales, the protective range argument), SmoothAttention, and the kernel section — compute-aware weight reordering and the register-level reordering — for mechanism. Skim the accuracy tables; read the throughput tables while asking which gain belongs to the quantization algorithm and which to the kernel, because the paper's answer to seed 3 is spread across both. Skip most of the ablation prose, none of the systems design.

Optional — SmoothQuant. Introduction and the migration formula (the α-balanced per-channel scale) only — ten minutes, read as the algebraic parent of both AWQ's weight-side scaling and QServe's SmoothAttention.

Optional — SpinQuant. The computational-invariance setup and the variance-of-random-rotations motivation for learning R; skim the results. Hold: what does W4A4 need that no scaling method can provide?

Looking ahead

Quantization shrinks the bytes per pass over the weights; speculative decoding (Wednesday, Nov 11) shrinks the passes per token, and the two multiply against the same 1.04 ms floor — including the question of what precision the draft model should run at, which is today's arithmetic recursed. Routing and load balancing (Oct 26) already took up the setting where a fleet mixing quantized and full-precision replicas becomes a routing problem with a quality dimension. Nov 18 and Nov 23 are the two agent-serving meetings (Lectures 18–22 across them): agent traffic is prefill-heavy (Lecture 5's ≈60× ratio), so today's 2×2 says W4 buys agents little while A8 and KV4 buy a lot — audit which row your system occupies before you quantize anything. The final project was announced Oct 26 and its proposal is due tonight: if yours touches prefix reuse, note that KV4's 4× capacity multiplies whatever hit rate your strategy earns — bytes saved per entry and entries reused are independent multipliers. And if you are choosing where to spend Assignment 5 (optimize the full stack, due Dec 2): a measurement that confirms or kills §15.5's four-instruction board model on a real kernel — Nsight in hand, issue-slot utilization as the verdict — is explicitly a good investigation.