For eleven weeks the arrow has pointed one way: agents were the workload and systems were the thing we built for them. Today it reverses, and the honest framing is that this is the least settled material in the course — so the emphasis is on how to evaluate a claim rather than on results that will age badly. The thesis is narrow and defensible: an agent's usefulness on a task is governed by the cost of checking its work, and systems tasks are unusually blessed with cheap, legible, non-gameable checkers that were built for other reasons and were never designed to be fooled. Compilers, test suites, and benchmarks are exactly the verifiers Lecture 5 §5.2 priced at turning a 36% task into an 82% one for 25% more wall-clock. That is why the first credible results in this area are in software engineering and GPU kernels rather than anywhere else, and it is why the interface — not the model — has been the dominant variable. By the end you should be able to say what makes a systems task tractable, explain why an agent-computer interface moved SWE-bench scores more than model choice did, read a two-part benchmark metric like fast_p correctly, and apply a measurement checklist that most papers in this literature currently fail.
Part I built the vocabulary: an agent is a control loop around a stateless model (Lecture 3), the loop and its tool interfaces are yours to design (Lecture 4), and a change is only a change if a frozen task set says so (Lecture 5). Part II spent eleven meetings making the machine underneath it efficient, and ended by making the program legible to the scheduler (Nov 18, Nov 23).
Part III asks the reverse question, and the connective tissue is not thematic but technical: every tool Part I gave you for judging an agent is exactly the tool needed to judge these claims. Lecture 5 §5.2's verifier arithmetic explains which tasks are tractable; §5.3's frozen set explains why most reported numbers are softer than they look; §5.5's diagnose-in-order rule explains why interface work beat model work. Its companion note, LLM for Systems Research II, takes the next step up — agents attempting research rather than tasks — and today is the groundwork for it.
What today is not. It is not a survey of model capabilities, and nothing here should be read as a claim about what models can do in general. Capability results in this area have a half-life measured in months, so the durable content is the evaluation apparatus: what a task needs in order to be attemptable, what a benchmark measures and misses, and which numbers a paper owes you. Those outlast any particular model, and they are what the self-check tests.
75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221). Instructor-led, two required papers, opening Part III. The temptation is to spend the hour on capability anecdotes; resist it. The two protected segments are both about evaluation.
| Time | Segment | Notes |
|---|---|---|
| 0–7 | §23.1 The arrow reverses | Say plainly that this is the least settled material, and why that changes how we read it. |
| 7–20 | §23.2 Verifiers decide | Protected — never cut. Re-derive 0.36 → 0.82 at +25%. This is the lecture's thesis. |
| 20–34 | §23.3 SWE-agent's interface | The ablation is the paper. Map each decision to a Lecture 4 §4.2 principle. |
| 34–46 | §23.4 KernelBench and fast_p | Why two-part metrics. Then the pass@k inflation arithmetic. |
| 46–55 | §23.5 Why kernels are the ideal testbed | Reuse Lecture 9's numbers as the task. The 768-config search space. |
| 55–65 | §23.6 What benchmarks miss | Contamination, pass ≠ correct, one-shape kernels. |
| 65–72 | §23.7 The checklist | Protected. Five questions to ask every paper in Part III. |
| 72–75 | §23.8 Wrap | Point at the companion note. End on the verifier line. |
Reading-only, not scheduled: §23.8's agenda in full, and the BountyBench/multimodal material.
If running long: compress §23.5 to the search-space number and drop §23.6's third paragraph. Never cut §23.2 or §23.7 — the verifier arithmetic is why this material belongs in this course rather than in a general AI course, and the checklist is what students will actually reuse on the companion note.
By the end of this class you should be able to:
fast_p is a two-part metric and what a single-number version would hide.k inflates an apparent success rate, and convert it into cost per completed task.Part II's question was "how do we serve this workload efficiently?", and it had the property that made it teachable: the answers are arithmetic. A ridge point is 295 FLOP/byte whether or not you like it, and a 25.6× cache reduction is a division. Today's question — "can a language model do systems work?" — has no such property. It is empirical, the evidence is a few years old, and the underlying capability is moving faster than the literature evaluating it.
Two consequences shape the rest of Part III.
Results will age; methods will not. Any number in the assigned papers about what fraction of tasks a model solves is a statement about a particular model on a particular set on a particular date, and will be wrong within a year in the optimistic direction. What will still be true is the structure of the question: which tasks admit cheap checking, what an interface is worth, what a benchmark's metric conceals. So the durable content is evaluation, and that is what today is about.
This course is unusually well equipped for it. Not because we know about models — we have deliberately never asked whether a model is good (Lecture 1 §1.1) — but because Part I built the measurement apparatus. Lecture 5 spent a full meeting on frozen task sets, cost per completed task, sample sizes, and the discipline of changing one thing at a time. That apparatus is exactly what this literature needs and frequently lacks, and applying it is a contribution a student in this room can make.
Minutes: 7. Board: "Part II: arithmetic. Part III: evidence." Then "results age, methods do not." Ask the room: "Which lecture in Part I is the most useful thing you own for reading these papers?" Lecture 5. Make them say it. Expect confusion: Students expect a capabilities lecture. Set the frame explicitly in the first two minutes or you will fight it for the rest of the hour.
This is the thesis, and it is Lecture 5 §5.2 restated as a claim about task selection rather than about agent design.
An agent working a multi-step task compounds its per-step failure rate. Lecture 5's arithmetic: at 95% per step, a 20-step task succeeds 0.95²⁰ = 36% of the time. Insert a verifier inside the loop that catches a fraction c of errors, and the effective per-step reliability becomes q' = 1 − (1 − q)(1 − c); at c = 0.8 that is 0.99 per step, so 0.99²⁰ = 82%, bought for about 25% more wall-clock. A single component more than doubles the completion rate.
Now read that as a statement about which problems to point an agent at. The multiplier depends on having a verifier that is cheap (you can afford to run it every step), legible (its output tells the model what to fix, or the retry fails identically), and non-gameable (passing it means the work is right, not that the check was defeated). Lecture 5 named those three properties for an agent you build; the same three decide whether a domain is attemptable at all.
And here is why systems is the domain where this landed first. Its verifiers already exist, and they were not built to be gamed. A compiler rejects your program with a line number. A test suite fails with an assertion and a stack trace. A benchmark returns a number in microseconds. Each of these is cheap, extremely legible, and — crucially — was designed decades ago by people who had no idea it would one day serve as a reward signal, which is the best possible guarantee that it is not measuring something optimized-for.
| Verifier | Cheap? | Legible? | Non-gameable? |
|---|---|---|---|
| Type checker / compiler | ~1 s | excellent — line, column, expected type | yes, within its scope |
| Unit test suite | seconds | good — assertion and trace | no — tests can be edited, or special-cased |
| Kernel correctness check vs a reference | ~1 s | moderate — a numerical mismatch, not a cause | yes — the reference is not yours to change |
| Wall-clock benchmark | seconds | poor — "slower", with no reason | yes, if the harness is trusted |
Note the pattern: the two with the best legibility are the two with a gameability or scope problem, and the two that are hardest to fool tell the model least about why it failed. No single verifier has all three properties, which is why the working recipe is a ladder — type check, then tests, then build, then benchmark — exactly as Lecture 5 §5.2 proposed.
The corollary is the useful part for research taste. Where the verifier is weak, results should be distrusted in proportion. Ask of any claim in this area: what checked the work, what would it accept that a human would reject, and could the agent have optimized against it? That question is the reason §23.6 exists and the reason SWE-bench Multimodal is on the reading list — it is what happens when the test suite is taken away.
Minutes: 13. Protected. Board: 0.95²⁰ = 0.36, then 0.99²⁰ = 0.82, then "+25%". Then the three properties. Then the four-verifier table, filling the last column last. Ask the room: "Which of these four could an agent defeat?" The test suite — by editing tests. Let them find it; it is the whole of §23.6's contamination worry in miniature. Expect confusion: Students hear "verifier" and think "evaluation". It is inside the loop, not after it. Lecture 5 §5.2's placement argument is worth thirty seconds.
SWE-agent's result is the one that should most interest a systems audience, and it is not a score. It is that holding the model fixed and changing the interface moved performance more than changing the model did. That is a systems finding about a machine-learning system, and it is the same claim Lecture 4 §4.2 made from first principles — the model re-reads your tool documentation on every call, with no compiler to catch a misunderstanding, so the schema is the program — now with a benchmark behind it.
Their agent-computer interface makes four decisions worth naming, and each maps onto a Lecture 4 principle.
A file viewer with a bounded window rather than "here is the file". Lecture 4 §4.2's output cap, enforced in the tool rather than in the prompt — and Lecture 4 priced the alternative: one uncapped 20,000-token dump at step 4 of a 20-step task rides in 16 subsequent prompts for 320,000 tokens, more than three times the whole task's budget.
An editor that lints on every write and rejects a syntactically broken edit rather than accepting it and failing later. This is Lecture 4 §4.5's errors as data, plus Lecture 5 §5.2's legibility: the model is told immediately, in a form it can act on, at the moment the context still contains what it was trying to do. A broken edit accepted now becomes an inscrutable test failure ten steps later.
A search that returns a summary with locations rather than every match. Context is the scarce resource (Lecture 3 §3.5), and a search that floods it has spent the task's budget to answer one question.
A small, closed command set rather than a general shell. Fewer ways to be wrong, and every command's output shaped deliberately — which is Lecture 4's argument that a small agent you fully understand beats a capable one you assembled.
The reason to spend class time on this is not the individual choices; it is what the ablation implies about where effort should go. The interface is the part you control, it is cheap to iterate on, and it was worth more than waiting for a better model. For a systems audience that is a familiar shape — it is the same reason Lecture 2 §2.9 said PyTorch beat TensorFlow by optimizing the human rather than the throughput — and it is directly actionable in Assignment 5.
Minutes: 14. Board: Four ACI decisions in a column; beside each, the Lecture 4 or 5 section it instantiates. Ask the room: "Which of these did you already build into your Assignment 2 agent?" Then: "which one would you add now?" The second question is the useful one. Expect confusion: Students read the ACI as ergonomics. It is a token-budget and legibility intervention, and both are quantitative. Use the 320,000-token figure.
KernelBench asks a model to write a GPU kernel for a specified operation, then checks it two ways: correctness against a reference implementation on test inputs, and speed against a baseline. Its headline metric is deliberately two-part.
That shape is the right one, and it is worth being explicit about what each single-number alternative would have concealed. Correctness alone rewards emitting a wrapper around the reference implementation — correct, zero speedup, and arguably the highest-scoring degenerate strategy available. Speed alone rewards a kernel that returns garbage very quickly, which is the classic reward-hacking failure. Mean speedup over correct kernels hides how few were correct, and is dominated by outliers. Only the conjunction, swept over p, describes the frontier — and sweeping p is what turns a metric into a curve, so fast_1 (any correct speedup) and fast_2 (a real one) can be reported separately and tell different stories.
There is a second measurement issue in this literature that KernelBench's structure makes visible, and it is Lecture 5 §5.3's arithmetic in a new costume: pass@k inflation.
k does to an apparent success rateLet p be the per-attempt success probability. Independent attempts give pass@k = 1 − (1 − p)^k.
At p = 0.20: pass@1 = 20%, pass@5 = 67%, pass@10 = 89%.
So the same agent can be described as solving one task in five or nine in ten, depending on a reporting choice. Now price it in Lecture 5 §5.3's currency — cost per completed task: at p = 0.2 you spend 1/0.2 = 5 attempts per success, and reporting pass@10 = 89% obscures that you paid for 10 and completed 0.89, i.e. 11.2 attempts per completed task.
Compare Lecture 5's own numbers, which are the same phenomenon: 22.3 calls per completed task against 15.6 attempted and 9 that a demo shows — a factor of 2.5× between the honest figure and the demonstrated one.
The general point, which applies to every paper in Part III: pass@k for k > 1 is a statement about a sampling budget, not about capability, and it is only interpretable alongside the per-attempt rate and the cost. It is the benchmark-scale version of the "large language monkeys" observation, and the reason discussion seed 3 asks you to convert every reported number into cost per completed task before comparing anything.
Minutes: 12. Board: fast_p's definition, then three crossed-out alternatives with their degenerate strategies. Then 20% / 67% / 89% and "11.2 attempts per completed task". Ask the room: "What is the highest-scoring cheat against correctness-only?" Wrap the reference. Getting there in ten seconds is a good sign about the room. Expect confusion: pass@k is read as capability. It is a budget. Say it twice and put the cost figure beside it.
Of all the tasks one could pose, GPU kernel generation is unusually well suited, and the reasons are worth enumerating because they double as a recipe for constructing a good task in any domain.
Both halves of the objective are machine-checkable, and neither is a proxy. Correctness is a numerical comparison against a reference; speed is a measurement in microseconds. There is no rubric, no judge model, and no human in the loop — which removes the largest source of noise in agent evaluation and the largest opportunity for gaming.
The search space is large but structured. This is the part that makes it a search problem rather than a knowledge problem, and it is worth a number.
A Triton matmul kernel's autotuning space, on the conventions of Sep 28: BLOCK_M, BLOCK_N, BLOCK_K each from {32, 64, 128, 256}; num_warps from {2, 4, 8}; num_stages from {2, 3, 4, 5}.
4 × 4 × 4 × 3 × 4 = 768 configurations — for one kernel, at one shape, before any change to the algorithm.
And the algorithmic space above it is not enumerable at all: Lecture 9's four ideas — tiling, online softmax, recomputation in the backward pass, split-K — are structural rewrites, not parameter choices.
The reward is dense and immediate. A wrong kernel fails in a second; a slow one reports a number. Contrast a task whose signal arrives only after a full training run, or after a human reviews a design document — Lecture 5 §5.2's compounding arithmetic simply does not apply when the verifier is expensive.
And, for this room, every task is one you have done. Lecture 9 is a catalogue of candidate problems. An agent asked to write a fast attention kernel is searching for exactly the transition that lecture derived by hand: unfused attention sits at 64 FLOP/byte and 22% of an H100's peak, and keeping the score matrix off HBM moves intensity to S/2, crossing the ridge at 590 tokens — a 4.6× on the kernel. That is a well-posed, machine-checkable, non-obvious optimization with a known answer, which makes it an unusually informative test: you can ask not merely whether the agent got a speedup but whether it found that speedup, and if not, which of the four ideas it missed.
The honest counterpoint, and it is the reason §23.6 follows immediately: a kernel that is correct and fast on the benchmark's shapes may be neither on yours. Lecture 9 §9.5 made exactly this point about FlashAttention's own gains — 4.6× on the kernel, 1.18× on a 4K prefill, 1.73× at 32K — so a kernel-level result is not a deployment result, and a benchmark that fixes the shapes cannot tell you which you are getting.
Minutes: 9. Board: 768, then "and the algorithmic space is not enumerable". Then Lecture 9's 64 → S/2 → 590. Ask the room: "Which of Lecture 9's four ideas would a model find?" Tiling and split-K are in the literature it trained on; online softmax as a derivation is the interesting case. There is no settled answer — that is the point of the question.
Four gaps, in rough order of how much they should change your reading.
Contamination, and the frozen-set problem. SWE-bench tasks are drawn from public repositories whose issues, discussions and fixes are in training corpora. A model may have seen the patch. This is not a hypothetical objection — it is the reason held-out and freshly-collected variants keep being constructed — and it is Lecture 5 §5.3's frozen-set discipline colliding with the fact that you cannot freeze a set against a model whose training data you do not control. The best available responses are tasks created after a model's cutoff, tasks whose solutions were never public, and reporting on both. None is fully satisfying.
Passing the tests is not being correct. A patch that satisfies a repository's test suite may be wrong in ways the suite never checked — that is what it means for a suite to be incomplete, and every suite is. Worse, the agent can sometimes reach the tests themselves, at which point the verifier's non-gameability (§23.2) fails outright. The standard mitigation is to hide the tests and forbid editing them, which is exactly an admission that the verifier only works when the agent cannot reach it.
No signal on the qualities engineers actually argue about. Nothing in a pass/fail harness measures whether a patch is maintainable, whether it fits the codebase's conventions, whether its abstraction is right, or whether a reviewer would accept it. Those are the properties that dominate real engineering time, and they are absent by construction because they are not cheaply checkable — which is §23.2's thesis biting back: the tasks that got attempted first are the tasks whose checkable part is the whole task, and that is a selection effect, not a discovery about difficulty.
Single-shape, single-machine kernel results. As §23.5 closed: correct and fast at the benchmark's shapes and on the benchmark's GPU. A kernel tuned for one shape can be slower than the baseline at another, and Lecture 9's Amdahl arithmetic means even a genuine 4.6× kernel win is a 1.18× end-to-end win at 4K. A benchmark reporting kernel-level speedups is reporting the larger of the two numbers.
Minutes: 10. Board: Four bullets. Under the third, write "selection effect, not a discovery". Ask the room: "If you had to fix one of these, which?" Contamination is the one with no clean answer; the maintainability gap is the one that matters most in practice. Expect confusion: Students treat contamination as a solved problem because "they held out a test set". Holding out from your training run does not hold out from the model's.
This is the part to keep. Ousterhout's Always measure one level deeper argues that the most common measurement failure is stopping at the top-level number and never asking what produced it, and this literature currently fails that test routinely. Five questions, in the order that catches the most.
1. What was the verifier, and could the agent reach it? §23.2's three properties. If the check was a test suite the agent could read or edit, the result is about the harness.
2. What is the per-attempt rate, and what is the cost per completed task? §23.4. A pass@k with no k, no per-attempt rate, and no token or GPU cost is uninterpretable — and Lecture 5's 22.3-against-9 gap says the honest number is typically 2–3× the demonstrated one.
3. What is the interval, and how many tasks? Lecture 5 §5.3: 20 tasks gives ±20 points at p ≈ 0.7, and halving that interval costs 4× the tasks. Most reported differences in this area are smaller than their own error bars, and the paper usually does not say. A 2-point improvement on a 50-task suite is not a result.
4. What was the baseline, and was it configured competently? This course's own running example: Lecture 12 Exercise 5 showed a 4.5× disaggregation claim can be manufactured entirely by leaving chunked prefill off in the baseline. The agent equivalent is comparing against a bare model with no interface, when §23.3 says the interface was the variable — so "our agent beats the raw model" measures the ACI, not the agent.
5. Where did the time and money go? Lecture 5 §5.4's invariant, model + tool + gap = wall-clock, and the token ledger that must sum to the input. If a paper reports a success rate and no cost, it has reported the numerator of a fraction.
"Our agent resolves 43% of SWE-bench Verified, up from 33%, with GPT-class model X."
(1) Verifier: repository test suites, hidden from the agent — good, though contamination remains. (2) Per-attempt vs pass@k: if 43% is pass@1, fine; if pass@5, the per-attempt rate is nearer 11% and the cost is ~9 attempts per completed task. The paper must say which. (3) SWE-bench Verified is 500 tasks, so a 43% estimate carries about ±4.3 points at 95% — meaning a 10-point gain is real and a 3-point gain would not be. Sample size rescues this one. (4) Baseline "33%": same interface? Same scaffold? If the 33% is a different scaffold, the comparison is of scaffolds, not of the change claimed. (5) Cost: absent from most such claims. At 20 steps and Lecture 3's ledger, a single attempt is order $0.18–$0.55, so 500 tasks × 5 attempts is a real number the reader should be told.
Minutes: 7. Protected. Board: The five questions, numbered. They are the rubric for the companion note as well. Ask the room: "Which of the five does the required reading answer?" SWE-agent answers 1 and 4 well (the ablations are the baseline discipline) and 5 poorly.
Reading-only; not scheduled in class.
Stated as open questions rather than predictions, and deliberately in the terms this course has built.
Which systems tasks have cheap verifiers that nobody has wired up yet? §23.2 says this is the question that determines what becomes attemptable. Candidates from Part II: an eviction policy scored on a replayed trace, which the parked competition page still specifies and which is therefore a task an agent could attempt; a scheduling policy scored on goodput in a simulator; a kernel scored by fast_p. Each has a machine-checkable objective and a large structured search space.
Can an agent do the analysis rather than the search? Lecture 9's win came from writing FLOPs in one column and bytes in the other and noticing a matrix nobody wanted. That is a different act from sampling 768 configurations, and it is what the course has been training humans to do. Whether it is reachable by a model is genuinely open and is the most interesting question on this list.
How do you evaluate a change when the verifier is the expensive part? All of today's tractable tasks have sub-second checks. The tasks that matter in systems research often have checks measured in GPU-hours — does this optimization help on real traffic? — and Lecture 5 §5.2's compounding arithmetic collapses when the verifier cannot run every step.
What is the cost per accepted contribution, and is it falling? Question 2 of the checklist, asked of the field rather than of a paper. It is the only number that would settle whether this is a research direction or a product.
fast_p is two-part on purpose. Correctness alone rewards wrapping the reference; speed alone rewards fast garbage; mean speedup over correct kernels hides the correctness rate. Only the conjunction, swept over p, describes a frontier.k is a budget, not a capability. At a 20% per-attempt rate, pass@1 = 20% and pass@10 = 89% — the same agent — and the honest figure is 11.2 attempts per completed task. Lecture 5's own 22.3-against-9 gap is the same phenomenon at 2.5×.| Quantity | Value | Source |
|---|---|---|
| Compounding without a verifier | 0.95²⁰ = 36% | Lecture 5 §5.2 |
| With a verifier catching 80% | 0.99²⁰ = 82%, at +25% wall-clock | q' = 1−(1−q)(1−c) |
| Uncapped tool output, step 4 of 20 | 320,000 prompt tokens | Lecture 4 §4.2 |
pass@1 vs pass@10 at p = 0.2 | 20% vs 89% — same agent | 1 − (1−p)^k |
Cost per completed task at p = 0.2, k = 10 | 11.2 attempts | 10 ÷ 0.89 |
| Lecture 5's demo gap | 22.3 vs 15.6 vs 9 calls (2.5×) | Lecture 5 §5.3 |
| Interval on 20 tasks | ±20 points; 4× tasks to halve it | Lecture 5 §5.3 |
| Triton matmul tuning space | 768 configs, one kernel, one shape | §23.5 |
| The kernel task, from Lecture 9 | 64 FLOP/byte → S/2; ridge at 590 tokens; 4.6× | Lecture 9 §9.2 |
| Kernel win vs end-to-end win | 4.6× vs 1.18× at 4K, 1.73× at 32K | Lecture 9 §9.5 |
fast_p two-part, and what does each single-number alternative reward?Because correctness and speed can each be satisfied degenerately. Correctness alone rewards wrapping the reference implementation — correct, zero speedup. Speed alone rewards returning garbage quickly. Mean speedup over the correct subset hides how small that subset is and is outlier-dominated. Only "correct and at least p× faster", swept over p, describes a frontier rather than a point — and reporting fast_1 and fast_2 separately tells you whether the speedups are real.k is a statement about budget, not capability, and it is uninterpretable without the per-attempt rate and the cost. Lecture 5's 22.3-vs-9 gap is the same inflation seen from inside a single agent.admit(entry) and evict() -> victim, a replay(trace, capacity) -> score command returning recovered-prefill under the competition metric, and a read-only trace summary (length distribution, reuse-distance histogram) — deliberately not the raw trace, since dumping it floods context (Lecture 3 §3.5). Budget: 30 steps, one replay per step at a few seconds each. Verifier: replay on a held-out trace slice — cheap (seconds), poorly legible (a scalar, no reason), non-gameable if the held-out slice is hidden. Expected failures: (a) overfitting to the visible slice — the classic gameable-verifier outcome, caught by the held-out split; (b) proposing LRU with cosmetic changes, since it is the strongest thing in the training data (Lecture 13 §13.8 says LRU is a defensible baseline, so this scores respectably and teaches nothing); (c) a policy that ignores the tree constraint and evicts interior nodes, which is a correctness bug the replay harness must reject explicitly. The interesting design choice is adding a legible second verifier — a diagnostic that reports hit rate by prefix length — converting a scalar into a signal the agent can act on, which by §23.2's arithmetic is worth more than any prompt change.p = 1 − (1 − 0.71)^(1/5) = 1 − 0.29^0.2 = 1 − 0.7788 = 22.1%; interval on the pass@5 estimate is 1.96√(0.71·0.29/100) = ±8.9 points, and the cost is 5 attempts per task for 0.71 completions = 7.0 attempts per completed task. C: ±1.96√(0.58·0.42/50) = ±13.7 points. Ranking: A (tight interval, honest pass@1), then B (usable once converted, but its headline is 3.2× its per-attempt rate), then C — whose ±13.7 points means it cannot distinguish itself from anything within 27 points. A and C's headline numbers differ by 15 points and C's interval swallows it, which is exactly Lecture 5 §5.3's "+2 of 20 is a coin flip" at benchmark scale.p = 0.43 needs the interval on the difference below 5 points: 1.96√(2·0.43·0.57/n) ≤ 0.05 → √(0.49/n) ≤ 0.0255 → n ≥ 754 per arm, 1,508 runs. Paired (same tasks both arms) removes task difficulty as a variance source; with a per-task McNemar-style comparison and a plausible discordance rate of 20%, n ≈ 300 tasks run twice, 600 runs. Cost at Lecture 3's ≈$0.18–$0.55 per 10–20-step session: 600 runs × $0.35 ≈ $210, plus harness time. The experiment costs a couple of hundred dollars and is almost never run — which is the exercise's point, and the reason §23.7's question 3 catches so much. Pairing is worth 2.5× here and is free.fast_2 — write the highest-scoring useless strategy, and say which metric survives. Solution sketch: Correctness only: emit return torch.matmul(a, b) — a wrapper around the reference. Scores 100%, speedup 1.0×. Speedup only: return an uninitialized tensor of the right shape; arbitrarily fast, always wrong. Mean speedup over correct kernels: solve one trivial problem with a 50× win and fail everything else; mean speedup 50×, coverage 1/n. fast_2: requires correct and ≥2×, so wrappers score 0 (1.0× < 2), garbage scores 0 (incorrect), and the one-lucky-kernel strategy scores 1/n — honestly. fast_2 survives all three attacks. The residual attack on fast_2 is shape-specific tuning: hard-code the benchmark's dimensions, which is correct and fast on the suite and useless in deployment — defeated only by evaluating on unseen shapes, which is §23.6's fourth gap and the one no current benchmark fully closes.fast_p at several p — good. (3) Task counts per level are modest, so per-level differences carry wide intervals. (4) Baseline is PyTorch eager/compiled — reasonable and clearly stated. (5) Strongest point: fast_p itself, which forecloses the three degenerate strategies of Exercise 4. Weakest: single-shape, single-GPU evaluation, so a benchmark result does not transfer to a deployment — the gap Lecture 9 §9.5 quantified as 4.6× against 1.18×.Required — SWE-agent. Read the ACI section first and the ablations second, and treat the ablations as the contribution: they are the part that establishes interface rather than model as the variable, and they are a clean instance of Lecture 5 §5.5's change-one-thing discipline. As you read each interface decision, write down which Lecture 4 principle it instantiates — bounded windows are the output cap, lint-on-write is errors-as-data plus legibility, summarizing search is context-as-scarce-resource, and the closed command set is the small-agent argument. Skim the benchmark-scores discussion; treat absolute numbers as dated. Hold this question: which of their decisions did you make by accident in Assignment 2, which did you get wrong, and what would you change in Assignment 5 tomorrow?
Required — KernelBench. Read the task construction and level structure, then spend real time on the metric: be able to say what fast_p forecloses that each single-number alternative permits (Exercise 4 is that list). Read the failure analysis carefully, since it is the most information-dense part for a systems reader — the kinds of mistakes tell you what the search is and is not exploring. Then read it against Lecture 9: pick two problems and ask which of tiling, online softmax, recomputation, or split-K a solution would need. Hold this question: the benchmark fixes shapes and hardware — write down what you would need to add to make a result predictive of a deployment, and estimate what that would cost to run.
Optional — LLMs for compiler optimization. The same bet with a compiler as the verifier and instruction count as the objective. Question: the verifier here is more legible than a wall-clock benchmark and less gameable than a test suite — does that predict better results, and does the paper bear it out?
Optional — OpenHands. The interface question as a platform. Read if you are building tooling for Assignment 5. Question: which of SWE-agent's four decisions does it generalize, and which does it abandon?
Optional — SWE-bench Multimodal. The sharpest instrument on this list for §23.2's thesis, because it changes the verifier and holds much else fixed. Question: how much of current performance is attributable to the verifier rather than the model — and what does your answer imply about which domains will see results next?
Optional — BountyBench. Tasks with real stakes and adversarial structure, where a successful exploit verifies itself. Question: self-verifying tasks have perfect non-gameability — why has that not made them the easiest domain?
Optional — Always measure one level deeper. The most useful reading on the list and the shortest. §23.7 is this paper applied to this literature. Question: Ousterhout's central complaint is stopping at the top-level number — find the top-level number in each required reading and say what is one level beneath it.
LLM for Systems Research II — the companion note, also optional — moves up a level: from agents doing tasks with cheap verifiers to agents attempting research — MLGym and AI Scientist-v2 — where the verifier is expensive, contested, or human. §23.7's five questions are that meeting's rubric, and §23.8's third open question is its central difficulty: Lecture 5's compounding arithmetic simply does not apply when the check cannot be run every step. Come having decided what you think a credible claim would even look like there.
Assignment 5 is due Dec 2, and the final project — announced Oct 26, proposal due Oct 28, report due Dec 8 — is where today's material is most directly useful: a project that builds a tool has to build the verifier that says whether the tool earned its time, which is §23.2's whole argument turned into a deliverable.
One thing to carry out of the room. The reason this material belongs in a systems course is not that agents write code. It is that the binding constraint on what an agent can do is the cost and quality of the check, and building cheap, legible, non-gameable checks is a systems-design problem — the same problem, in a different costume, as the one Lecture 5 set you in September. Everything in Part III is downstream of it.