Assignment 5 — Optimize the full stack (18%)

Out Wed Nov 11 · due Wed Dec 2, 11:59pm · individual · all five assignments

Assignment 4 got the quality back. Freeze that recovered quality as a threshold before tuning; now hold it still and make the system cheap — this time with the serving system in scope, which Assignment 3 did not have. You own the whole stack — the model, the engine serving it, and the loop driving it — so every axis is genuinely yours to move:

1. Optimize the model. Quantization, a smaller model, distillation.
2. Optimize the serving system. Batching, prefix and KV-cache reuse, scheduling, speculative decoding.
3. Optimize the harness. Fewer and cheaper calls, less re-sent context, earlier stopping, parallel tool use.

Pick the axes your own measurements justify — but touch at least one serving-side axis and the harness, so that you feel both halves of the problem. Three steps, in this order, and the order is part of the grade:

1. Profile the Assignment 4 system until every token and every second is accounted for. The five metrics tell you what the system costs; they do not tell you where it goes. Add the serving-side numbers that do: time to first token versus decode time per call, decode throughput in tokens per second, GPU-seconds per task, batch occupancy, and — once you start reusing context — prefix-cache hit rate. Prefill and decode are different bottlenecks with different fixes, and a profile that cannot separate them cannot choose an axis. State the load you measured under: a throughput number taken at unbounded queue depth is not a throughput number, it is a latency you did not report.
2. Predict, then lock it in. Before changing anything: which axis, why you expect it to be the biggest win, and roughly how much. Timestamped, not revisable.
3. Optimize and re-measure on the same task set, the same five metrics, same method.

This one is written as a blog post

Like Assignment 3, the write-up here is a published blog post contributed by pull request to the course blog repository — mechanics and the deadline rule are on the Assignment 3 page.

This is the last thing you write for the course and the only place the whole stack appears at once, so it is worth more care than the 4% suggests. The story runs from a profile of a system you own end to end, through a prediction you committed to in writing, to what the measurement said instead. You have both halves that Assignment 3 could not reach: the harness levers and the serving ones, which means you can say something most people writing about inference cost cannot — which layer the win actually came from.

Reuse freely from your Assignment 4 report, and from your Assignment 3 post if it helps: the quality you recovered in A4 is the threshold this whole assignment holds still, so the reader needs it. Say what you are reusing. The section that earns the most and gets cut the most often is what you tried that did not work — a prediction that was confidently wrong, written up honestly, is worth more here than another win.

How the 18% breaks down: the profile (3%); the locked prediction, judged on reasoning rather than on whether it turned out right (3%); measured improvement with quality held and reported honestly (8%); and the blog post, including what you tried that did not work (4%).