Last class you built a loop that runs. A loop that runs is not an agent that works, and the distance between the two is the rest of agent design: the specification the agent is held to, the verifier that catches a bad step while its error is still local, and the task set that tells you whether any of your changes helped. Those three are one subject, because each is worthless without the others — a specification nobody checks is a wish, a verifier with no task set is an anecdote, and a task set scoring an unstated criterion is a number that means nothing in October. By the end you should be able to write a specification a stranger could satisfy and your is_final could evaluate, place a bounded non-gameable verifier inside the loop and price it, build the harness that turns "it worked" into a measured success rate with a stated uncertainty, and decide from a run's evidence which of the four failure classes you are looking at.

Where this sits

Lecture 3 evaluated an agent from the outside: success rate, cost per completed task, the four failure classes. Lecture 4 handed you the loop and its four decisions — what enters the context, when to call a tool, when to stop, what to do on failure. Both left a gap, and Assignment 2 is standing in it. You now have a program with a dozen design decisions in it, no evidence about any of them, and a deadline. The question today is not "what else could an agent do" but the narrower and harder one: how do you find out whether a change to your agent made it better? Everything in this lecture exists to answer that, and the answer is not a model or a framework — it is three artifacts you write yourself, version alongside the code, and freeze before you start tuning.

The last five minutes turn the camera around one final time. The agent you have designed emits a request stream with a very particular shape, and Part II is fifteen meetings on the machine that has to serve it.

Instructor notes — Timing plan

75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221).

Time	Segment	Notes
0–4	Framing	"Your loop runs. How would you know if it got better?" Nothing goes out today; say so early.
4–16	§5.1 The specification you ship	Before/after on the board, then the 19× late-discovery number.
16–34	§5.2 Verifiers you build	First centerpiece. Derive q', then the 0.36 → 0.82 line, then the verifier's own token bill.
34–52	§5.3 The task set is the instrument	Second centerpiece. The ±20-point interval is the number of the day. Protect all 18 minutes.
52–60	§5.4 Instrumentation	The closing invariant is the whole segment; counters and plots are reading-only.
60–68	§5.5 Diagnose, then change one thing	The 14/20 → 16/20 coin-flip result lands here.
68–72	§5.6 More than one context	Brisk. The sub-agent return contract, then the crossover number.
72–75	§5.7 Assignment 2, and the bridge	Freeze the three artifacts. One slide on what Part II is for.

Reading-only, not scheduled: the counter list and log format in §5.4, the memory subsection in §5.6, and the whole of the Exercises. Say out loud that §5.4's counters are graded in Assignment 2 whether or not they were lectured.

Learning objectives

By the end of this class you should be able to:

1. Rewrite a one-line task request as a specification with a metric, a measurement procedure, an invariant, an out-of-scope list, a budget, and a done-condition — and say which line your loop's is_final actually reads.
2. Compute what a specification error costs as a function of the turn at which it is discovered, and explain why the cost is quadratic.
3. Place a verifier inside the loop, compute the per-step and end-to-end effect using q' = 1 − (1 − q)(1 − c), and price the verifier's own output against the transcript it rides in.
4. Name the three properties a verifier must have, and give a concrete way each one fails.
5. Build an evaluation harness: a frozen task set, a scoring criterion written before the first run, a versioned scorer, and a stated uncertainty on the success rate you report.
6. Say how many tasks or runs a claim of a given size needs, and recognise a result that is inside the noise.
7. Emit the counters that close model + tool + orchestration gap = wall-clock, and diagnose a failed run in the order tool → harness → specification → model.
8. Decide whether a piece of work belongs in a sub-agent, and state the return contract it owes the parent.

5.1 The specification is something you ship

Lecture 4 argued that a tool schema is documentation read fresh at every call by a caller who cannot ask a clarifying question and whose mistakes no compiler catches. The task specification is the same object one level up, and it has the same property: it is read once, literally, by a stranger. Lecture 3 named the failure this produces when the specification is thin — specification failure, the class where every component behaved correctly and the words underdetermined the intent, and the one that a successful-looking run does not announce.

The difference from last lecture, and the reason this is a design topic rather than a prompting topic, is where the specification lives. In your agent it is not something a user types. It is a file in your repository: the system prompt, the tool-use policy, the output contract, and the done-condition, all of which you version, diff, and — starting today — test. When you change it you have changed the system, and Assignment 2's report should be able to say which version produced which number.

A request as it is usually written:

Make the serving script faster.

The same request written so that a stranger could satisfy it, you could check that they had, and your loop could tell when to stop:

Goal: reduce median time-per-output-token on bench/trace_200.jsonl by at least 20%.
Measure with: scripts/bench.py --trace bench/trace_200.jsonl --repeat 5 --warmup 1
  (report median and min/max across the five runs; the harness prints all three)
Invariant: generated text must match the current output byte-for-byte at temperature 0.
Out of scope: changing the model, the tokenizer, or the sampling parameters.
Budget: stop after 40 tool calls or 20 minutes and report what you have.
Done when: the measured median improves by >= 20% with the invariant holding, or you
  can show why it cannot, with the profile that says so.

Six lines, six removed arguments: a metric replaces an adjective, a measurement procedure replaces "faster", an invariant states what must not change, an out-of-scope list forecloses the cheap wrong answers, a budget bounds the run, and a done-condition tells the loop when to stop instead of leaving is_final to guess. Two of those six are not addressed to the model at all. The budget is a loop condition — Lecture 4 §4.5's safety property, written where the agent can also see it so its plan and your cap agree. The done-condition is the predicate is_final evaluates, which is why a specification whose stopping rule is "when it looks good" produces an agent that cannot stop: you have asked a text generator to evaluate a criterion you never made checkable.

Front-loading is not tidiness. It is the cheapest point on a curve you can write down. Take Lecture 4's session shape — a 2,000-token pinned preamble, 300 tokens appended per turn — and let C(k) be the cumulative prompt tokens submitted through turn k:

And the tokens are the cheap part. A specification error discovered at turn 18 has also spent your attention, may have written to the world through a non-idempotent tool (Lecture 4 §4.5), and — worst — may not be discovered at all, because a fluent answer to the wrong question does not look like a failure. That is the entire argument for §5.2: the earlier a wrong step is caught, the less of the quadratic you have paid.

Instructor notes

Minutes: 12. Board: Put the one-line request up and ask the room for fixes; write them as a list. Then reveal the six-line version and tick off which of their suggestions it already contains — they will produce four of the six and almost never the budget or the done-condition. Then C(18) vs C(2) in the corner: 81,900 / 4,300 = 19×. Ask the room: "Which of these six lines is addressed to the model, and which to your own code?" The answer — budget and done-condition are both — is the point of the section. Expect confusion: Students treat this as prompt engineering, a matter of phrasing. Push back: the done-condition is a predicate your loop evaluates, and it is in the repository under version control. Common wrong answer: "Longer prompts are better prompts." No — checkable prompts are better prompts, and every extra token rides in every subsequent call at the rate C(k) describes. If short on time: Keep the metric/invariant/done-condition trio and the 19×.

5.2 A verifier is a component, not a habit

Lecture 3 established the compounding problem and it is worth restating in one line, because every design decision in this section is an answer to it: at 95% success per step, a 20-step task completes 0.95²⁰ ≈ 36% of the time, and errors do not merely accumulate — a wrong step enters the transcript and every later step reasons from it.

The fix is not a better model. It is a check that runs inside the loop, so a failed step is caught and repaired while its error is still local. Let q be the per-step success rate without a check, and c the fraction of step failures the check catches and the agent then repairs:

Where does it go? You have two placements and they are not equivalent. As a tool the agent may call, it is advisory: the model decides when to run it, which means it will skip it under time pressure or after a confident-looking step — exactly when it is needed. As a gate in invoke, run unconditionally after every step of a given kind, it is a property of the system: the result enters the transcript whether or not the model asked, and a failing check is a step the loop will not let stand. Serious agents use both — the ladder below as tools, the cheapest rung as a gate — but the decision is yours and it belongs in the design, not in the prompt.

Verifiers form a ladder, and the discipline is to climb only as far as the errors you actually make demand. A type or syntax check costs milliseconds and catches a narrow, very common class. Unit tests cost seconds. An integration test or a full build costs minutes. A benchmark costs minutes and returns a noisy number, which §5.3 treats as a measurement problem in its own right. You cost minutes to hours and are the rung that catches what none of the others can express.

Three properties decide whether a verifier helps or merely runs, and each fails in a specific way.

Legible. Lecture 4's "errors are data" applies here exactly. A check that fails with an empty diff, a bare exit code, or a 900-line stack trace tells the model nothing it can act on, and a failure the model cannot read is a step that will fail again identically on the retry. The output of a verifier is an interface, and SWE-agent's whole argument is that interfaces of this kind set the success rate.

Bounded. The verifier's output enters the transcript and is re-sent on every remaining call, so its cost is not what it prints — it is what it prints multiplied by how many prompts still carry it. This is Lecture 4 §4.2's output cap, and a per-step verifier is the worst case for it, because it fires at every step rather than once.

Non-gameable. This is the uncomfortable one, and it is a design property rather than a coding detail. The agent optimizes what you measure. A test that asserts nothing passes. A benchmark that reads a cached result gets faster without the code changing. A lint rule is satisfied by a suppression comment. Every weak verifier converts a visible failure into a confident one, which is strictly worse than having no verifier at all, because it also removes the evidence. The only general defence is a check the agent was never optimizing against — held out, run last, by you. This course is built on that principle: the later assignments score your system on held-out tasks it was never tuned against.

Instructor notes

Minutes: 18. First centerpiece — protect it. Board: q' = 1 − (1−q)(1−c) at the top. Under it 0.95²⁰ = 0.36 and 0.99²⁰ = 0.82, with "+25% wall-clock" beside them. Then the ladder as five rungs, vertical, with costs. Then 190·v and the two numbers 152,000 / 22,800 — derive the 190 in front of them, it is the same triangular sum as the context ledger and they should recognise it. Ask the room: "Would you rather have a model at 99% per step, or a two-second test suite the agent can run?" They are the same 0.82; one of them you can build this afternoon. Expect confusion: Verification is imagined as something done at the end, by a human. The entire point is that it belongs inside the loop, where a failure is still cheap and still local. Common wrong answer: "Add more tests." Ask what the tests assert, then tell the cached-benchmark story and let it land — a verifier the agent can satisfy without doing the work is this section's failure mode, and it is not hypothetical. If short on time: Keep the formula, the 0.36 → 0.82 line, and the non-gameable paragraph. The placement discussion can be read.

5.3 The task set is the instrument

Everything so far has been a change you might make. This section is about the only thing that can tell you whether a change was an improvement, and it is the part of agent design that gets skipped, because it produces no demo.

Start with what a single run is worth. Lecture 3 did this arithmetic and it deserves repeating in the designer's setting: an agent that succeeds 70% of the time demos flawlessly about seven times in ten, and ten clean runs are consistent, at 95% confidence, with a true failure rate of 26%. A run is a draw. Your agent's success rate is a parameter you can only estimate, and the estimate has an interval whether or not you report one.

Three artifacts make the instrument, and all three are frozen before you tune anything.

The task set. Twenty or more, varied enough that one lucky prompt cannot carry the score, and representative of what you actually want the agent to do. Fix it now. Every later comparison in this course — Assignment 3 driving cost down from outside, Assignment 4 taking the frontier model away, Assignment 5 driving cost down with the stack in scope — is a comparison against the numbers you produce on this set, and a set that drifts makes all of them meaningless. The drift is rarely deliberate: it is a task quietly reworded, or one dropped because it was "not really what the agent is for".

The success criterion, written down before the first run. A criterion written after you have seen the output is not a measurement; it is a description. This is the same non-gameability problem as §5.2 with you as the party being tempted, and the defence is the same — commit first, in a file, with a timestamp your version control can prove.

The scorer, versioned with the code. A script in the repository that reads a run and returns pass or fail, so the same tasks can be re-scored the same way in October. If scoring requires your judgement, the script's job is to present the evidence in a fixed format and record your verdict — the point is reproducibility, not automation for its own sake.

What to report is then decided, and Assignment 2 asks for exactly five numbers plus a table.

Metric	Report	Why this definition
Task success rate	Fraction passing the frozen criterion, count as well as percentage	A percentage without a denominator hides how much you know
Latency	Wall-clock per task, mean and P90	Agent latency is heavy-tailed; one task stalled in a retry loop hides inside the mean
Cost per task	Dollars at a price sheet you state once and keep	Prices move; a pinned rate keeps the number comparable across assignments
Tokens per task	Input and output separately, including retries and discarded output	Input and output differ ~5× in price and are fixed by different changes
Steps per task	Model calls and tool calls separately, mean and P90; a retry is a step	The harness's own number, and the one Assignments 3 and 5 attack directly

And the table that is not a number: failures broken down by cause — wrong answer, malformed tool call, ran out of steps, provider error, harness crash. The success rate tells you how often you failed; only the breakdown tells you what to fix. It also maps onto Lecture 3's four classes, which is what makes it actionable rather than descriptive.

One definition inside that table earns its own arithmetic, because the mean over attempted tasks is not the number you spend.

Two practical rules close the section. Judge on the set, never on the anecdote — a fixed set run several times, with the spread reported, is a morning's work and it is the difference between knowing your success rate and guessing it. And measure a no-op: run the harness against an unchanged agent and confirm it reports no change. If a no-op comes back as an improvement, your harness is caching something, and every number you have collected is about the harness rather than the agent.

Instructor notes

Minutes: 18. Second centerpiece. This is the material students most reliably skip and Assignment 2 most heavily grades. Board: √(p̂(1−p̂)/n) at the top, then 14/20 → 70% ± 20 points, boxed. Then "4× the tasks to halve the interval". Then the three frozen artifacts as three words: set / criterion / scorer. Finally 312 calls → 15.6 vs 22.3 vs 9, and circle the 9 as "what a demo shows". Ask the room: "Your agent passes 14 of 20. Mine passes 16 of 20. Is mine better?" Take the vote before §5.5 answers it — this is the setup for the coin-flip result. Expect confusion: Students believe more runs of the same task tighten the success-rate interval. They tighten the estimate for that task; the interval on the agent is set by the number of distinct tasks. Common wrong answer: "I will write the criterion after I see what it does." Name it: that is post-hoc scoring, and it is the same failure as the benchmark the agent games in §5.2, with you in the agent's role. If short on time: Keep the ±20-point interval and the three frozen artifacts. The five-metric table is on the assignment page and can be read.

5.4 Instrumenting your own agent

A success rate tells you whether. It never tells you where, and three measurements that no framework reports by default are what turn a score into a diagnosis.

Token attribution by component. Not a total — a breakdown: system prompt, tool schemas, tool outputs, verifier output, the model's own reasoning, and the tokens that were byte-identical to the previous call's prefix. The component that dominates is the one worth attacking, and it is almost never the one you assumed.

Latency by phase. Model time, tool time, and orchestration overhead, per step. Expect a surprise: in many agents the wall-clock is dominated by tool latency. A run with 20 model calls at 1.5 s and 15 tool calls at 2 s spends 30 s in each — round numbers, but a common shape, and it means halving model latency cuts end-to-end time by 25% at best, while issuing independent tool calls concurrently would do far more.

Model calls per completed task. §5.3's 22.3, reported as a distribution rather than a mean, because the mean describes neither the successes nor the failures.

This is the course's opening question — where do the time and the money actually go? — pointed at something you built, and it is Ousterhout's discipline: always measure one level deeper. "The agent took 60 seconds" is one level. "40 s of it was tool time, of which 30 s was one call we could have issued in parallel" is one level deeper, and only the second tells you what to change.

The counters, the log line, and the closing invariant

Three families of records, emitted by your own code, because nothing else will emit them.

Per model call: input tokens broken out by context component, plus the count that was byte-identical to the previous call's prefix. Output tokens. Wall-clock from request to last token, and time-to-first token separately if your client exposes it — TTFT is prefill and the remainder is decode (Lecture 2), and the two respond to entirely different fixes.

Per tool call: tool name, wall-clock, the terminal state from Lecture 4 §4.5's state machine, result size both before and after the output cap — the difference is what the cap bought you — and, if this call is a retry, the id of the call it retries. A verifier is a tool and gets the same record.

Per step and per task: the orchestration gap, wall-clock inside the loop that was neither a model call nor a tool call, where your own JSON parsing, transcript copying, and serialization hide. Per task: model calls issued, tokens in and out, wall-clock, and the terminal outcome — completed, failed, or budget-exhausted.

The invariant is what makes this a profile rather than a pile of numbers. For every task, model time + tool time + orchestration gap = wall-clock, and the token components sum to input tokens. If either fails to close, the remainder is unattributed cost — and unattributed cost is cost nobody optimizes.

One line per event, appended, never overwritten — Lecture 4's "the transcript is the state" applied to your measurements, so a crashed run still leaves a readable profile:

{"t":1789862444.11,"task":"t07","run":3,"step":4,"ev":"model","in":6412,"resent":5900,"out":81,"ms":1520,"by":{"preamble":2000,"task":180,"tool_out":3600,"reasoning":632}}
{"t":1789862446.24,"task":"t07","run":3,"step":4,"ev":"tool","name":"search_docs","ms":2130,"state":"ok","out_raw":4820,"out_capped":540}
{"t":1789862446.34,"task":"t07","run":3,"step":4,"ev":"gap","ms":95}
{"t":1789862471.02,"task":"t07","run":3,"ev":"task_end","steps":11,"in":71230,"out":940,"ms":41200,"state":"completed"}

The by field sums to in (2,000 + 180 + 3,600 + 632 = 6,412) — assert that in code rather than trusting it, because a component you forgot to attribute is a component you will not optimize. The tool line records the other quantity worth having: a 4,820-token result capped to 540, which is 4,280 tokens kept out of each of the seven remaining prompts, 29,960 in all.

Three plots are worth making from this file, and only three: input tokens by component against step index, stacked, which decides whether your lever is prefix reuse or a tighter output cap; a waterfall for one representative task, one bar per step segmented model / tool / gap, which decides whether to optimize the model at all; and the distribution of calls per completed task split by outcome, with the mean drawn on top so you can watch it describe neither group.

Instructor notes

Minutes: 8. Board: Three rows — tokens by component / latency by phase / calls per completed task. Then the 30 s + 30 s split, and ask what halving model latency buys. Then the closing invariant as an equation: "model + tool + gap = wall-clock". That equation is the segment; the counter list, the log format, and the three plots are reading-only. Ask the room: "In your Assignment 1 agent, what fraction of wall-clock was the model? Anyone actually know?" The silence is the point. Expect confusion: Latency is assumed to be model latency. Give the arithmetic and let them do the division themselves. Common wrong answer: "I will add logging later." The orchestration gap is the term nobody ever retrofits, and Assignment 3 grades exactly this file.

5.5 Diagnose, then change one thing

You have a failing run and a frozen task set. Lecture 3 gave the diagnostic order and the reason for it: tool, then harness, then specification, then model, because the first three leave evidence and the fourth is the explanation that requires no work. In your own agent each step of that order is now a question your instrumentation can answer.

• Tool. Did every call return what it claimed? Check the terminal states and the raw result sizes. Lecture 4's worst case is a tool that timed out and returned {"results": []}: "found nothing" and "did not run" are the same tokens, and only your log distinguishes them.
• Harness. Was the information the model needed actually in the failing call's context? This is a question about assemble, and you answer it by reading the prompt that was sent — not the one you believe was sent. Truncation and eviction are silent (Lecture 4 §4.3).
• Specification. Did the agent do what you asked rather than what you wanted? §5.1's failure. The tell is a run with no errors in it at all.
• Model. Everything needed was present, nothing was malformed, and the answer is still wrong. This is the diagnosis of last resort, and reaching for it first is how design work stops.

Then change one thing. Agents make this discipline harder than usual, because the tempting move is to change the system prompt, the tool schema, and the retry policy in one commit and re-run — and if the number moves you have learned nothing about which change moved it, while if it does not move you have possibly learned that two changes cancelled. Re-run the whole frozen set after each change, not the task that failed: a change that fixes one task and breaks two is a regression, and the only way to see it is to score the set.

Finally, know what your instrument can resolve. §5.3 gave the interval; here is the version that decides whether to keep a change.

The practical consequence is not to give up on small improvements — it is to notice that your first few changes should be large ones. Fixing an unbounded verifier output, adding a done-condition, or capping a tool that dumps 20,000 tokens moves numbers far outside the noise. Once you are down to effects of a few points, the honest reports are "no measurable difference on twenty tasks" and "here is what it would take to tell".

Instructor notes

Minutes: 8. Board: The four failure classes as a vertical list with an arrow down the side labelled "cheapest evidence first". Then the flip table: 3 up, 1 down, 16 unchanged, and 5/16 = 0.31 beside it. Return to the vote taken in §5.3 and show the room what it was worth. Ask the room: "Your agent failed. What is the first artifact you open?" Steer away from the answer "the final output" — Lecture 3's point that the answer is the least informative thing a run produces. Expect confusion: "Model failure" is reached for first because it requires no investigation. Insist on the order and on the evidence each step needs. Common wrong answer: "+2 out of 20 is a 10% improvement." It is two tasks. Do the flip arithmetic on the board and let it be uncomfortable. If short on time: The diagnostic order and the coin-flip number. The σ/√n line can be read.

5.6 When one context is not the design

Lecture 4 §4.4 introduced sub-agents and named their real payoff — context isolation, not parallelism. A sub-agent reads forty files and returns three lines, and the thirty-seven irrelevant results never enter the parent's transcript. Design-side, that payoff is bought by writing an interface, and the interface is the whole of the decision.

The return contract is the design. A sub-agent owes its parent a fixed, bounded, legible answer: what shape the result takes, what it says when it found nothing, what it says when it failed, and how many tokens it is allowed. A sub-agent that returns "here is everything I read" has isolated nothing — it has moved the parent's context problem one level down and then undone it on the way back. The three properties from §5.2 apply unchanged, because from the parent's point of view a sub-agent is a tool with an unusually expensive implementation.

And there is a crossover, which is arithmetic rather than taste. On k independent work items, with Lecture 4's shape — preamble P = 2,000, g = 300 appended per item, and a return summary of r = 100 tokens per worker:

One bad reason to fan out, and it is the common one: hoping several agents will repair a result that is wrong because the specification was thin or the verifier was weak. Fanning out a bad specification buys k confidently wrong answers, and §5.3's harness will tell you so if you let it.

Memory, and the design decision inside it

Everything above lives inside one session. The moment your agent should remember something across sessions you have a second design problem, and Lecture 3 named the trap: retrieval turns "the model does not know X" into an indexing problem, and MemGPT's virtual-memory analogy breaks at exactly one place — there is no page fault. Hardware faults because the access itself signals the miss. Your agent must decide to look something up, from a context that by construction lacks the information needed to know something is missing.

That makes memory a design decision rather than a feature to enable. Three questions settle it. What is worth persisting — durable facts about the task and the environment, or the whole transcript, which is usually a way of paying to re-read your own logs? What triggers a fetch, given that nothing will fault on your behalf — a rule in assemble that always loads a small pinned set, or a tool the model may call and will sometimes forget to? And what happens when the store is stale, since a confidently-cited out-of-date fact is exactly §5.2's confident failure with a citation attached. If you cannot answer the second question, pin the small set and skip the machinery; a pinned 200-token file beats a retrieval system nobody queries.

Instructor notes

Minutes: 4. Brisk — the arithmetic is a recall from Lecture 4, the return contract is the new idea. Board: "A sub-agent is a tool with an expensive implementation" and under it the four things its return contract must fix. Then k ≈ 6 with the two check rows. Ask the room: "Twenty files to summarize — one agent or twenty?" Vote, then run the arithmetic. Then ask "three files?" and let them find that the answer inverts. Expect confusion: Multi-agent is believed to be strictly better or strictly more expensive. It is neither; there is a crossover and it is computable. If short on time: The return contract and k ≈ 6. The memory subsection is reading-only.

5.7 What Assignment 2 has to produce

Nothing goes out today. Assignment 2 — design an agent is in flight: out Sep 21, due Sun Oct 4, 11:59pm, worth 10%, individual, and the assignment page has the spec. Just under two weeks remain, and this lecture is what the second half of that grade is for. Four percent of the ten is the five metrics and the failure breakdown, measured over a fixed task set with the criterion written down in advance — §5.3, exactly — and three more is the write-up, including the token and wall-clock accounting from §5.4 and an honest list of the questions you could not answer from outside the API.

Three things to do this week, in this order. Freeze the set, the criterion, and the scorer, before you tune anything, because every number in the rest of this course is measured against them. Build §5.4's counters into the loop rather than around it — the orchestration gap is the term nobody reconstructs afterwards. And run the no-op: score an unchanged agent twice and confirm your harness reports no difference.

The reason to be strict about the freeze is what comes next. Assignment 3 goes out Oct 5, the morning after this one is due, on this same agent: it holds success rate fixed and drives cost and latency down using only levers you wrote, because you are still renting the model. Assignment 4 then takes the frontier API away and asks you to recover the quality you lose, measured as the same five numbers on the same task set, and Assignment 5 drives cost down again with the serving stack in scope. All those comparisons are meaningless if the instrument moved in between.

Instructor notes

Minutes: 3, shared with the closing bridge. Board: Three dates: Sep 21 (A1 out), Sun Oct 4 → Mon Oct 5 (A1 due; A3 out), Sun Oct 25 (A3 due). Beside them the three words from §5.3: set / criterion / scorer, with "freeze" over the top. Ask the room: Nothing. Say plainly that a small agent with a real harness outscores an ambitious one with a demo, and that the harness is graded again in two later assignments. If short on time: Post the freeze instruction as an announcement and spend the time on the bridge — the bridge is what makes Monday make sense.

Key takeaways

Numbers worth memorizing

Quantity	Value	Where it comes from
Specification error at turn 2 vs turn 18	4,300 vs 81,900 tokens (19×)	C(k) = 2,000k + 150k(k−1)
Compounding without a verifier	0.95²⁰ = 0.36	20 steps at 95%
With a verifier catching 80%	0.99²⁰ = 0.82, +25% wall-clock	q' = 1 − (1−q)(1−c)
Uncapped vs capped verifier output	152,000 vs 22,800 added prompt tokens (6.7×)	190 · v, v = 800 vs 120
Success rate on 20 tasks	70% ± 20 points	2·√(p̂(1−p̂)/20)
Tasks to halve that interval	4× (20 → 80)	error falls as 1/√n
Calls per completed task	22.3 vs 15.6 attempted vs 9 demoed	14·9 + 6·31 = 312
"+2 out of 20"	p ≈ 0.31, no evidence	3 up, 1 down, (4+1)/16
Runs to resolve 3% at σ = 4%	8	2·4/√n ≤ 3
Fan-out crossover	k ≈ 6 items	3k² − 11k − 40 = 0
The session ledger everything is denominated in	97,000 submitted, 7,700 distinct	Lecture 4 §4.3

Self-check

Exercises

1. Price a verifier before you build it. Your agent runs 30 steps on a 2,500-token preamble, appending 400 tokens per step. You are considering a check after every step that emits v tokens. (a) Write the added prompt tokens as a function of v. (b) Evaluate it at v = 600 and v = 150, and compare each with the task's own cumulative ledger. (c) The check catches c = 0.7 of step failures against q = 0.96; give task success with and without it. (d) State the condition under which you would run the check every third step instead, and what it costs you. Solution sketch: (a) The step-k output rides in 30 − k later prompts, so the total is v · (0 + 1 + … + 29) = 435·v. (b) 435 · 600 = 261,000; 435 · 150 = 65,250; the task's own ledger is 30 · 2,500 + 400 · 435 = 75,000 + 174,000 = 249,000, so the uncapped check costs more than the entire task and the capped one about 26% of it. (c) q' = 1 − 0.04 · 0.3 = 0.988; 0.96³⁰ = 0.29 versus 0.988³⁰ = 0.70. (d) Only if the errors you are catching do not compound within three steps — otherwise you have reintroduced two steps of poisoned context between checks, and the saving is one third of 435·v while the loss is on the exponential.
2. Size the instrument to the claim. You want to be able to detect a change that moves success rate from 60% to 70%. (a) Using 2·√(p̂(1−p̂)/n) as the half-width, how many tasks does an independent comparison need before the two intervals stop overlapping? (b) Explain why a paired comparison on the same frozen task set needs far fewer, and what it is counting instead. (c) Your colleague proposes running the same 20 tasks 10 times each instead of building 200 tasks. What does that buy and what does it not? Solution sketch: (a) At p̂ ≈ 0.65 the half-width is 2·√(0.2275/n) = 0.954/√n; needing two half-widths to fit inside the 10-point gap gives 1.908/√n ≤ 0.10, so n ≥ 364 — independent comparison of proportions is brutally expensive. (b) Pairing removes the between-task variance: the 16 tasks that behave identically under both versions carry no information, and you count only the flips, so the test is on a handful of Bernoulli trials rather than on two noisy proportions. (c) Ten runs each tightens the estimate of per-task success and exposes run-to-run variance — genuinely useful, and it is how you learn whether your agent is nondeterministic. It does not narrow the interval on the agent's overall success rate, which is set by the number of distinct tasks.
3. Close the profile. A task reports wall-clock 84 s, model 26 s over 18 calls, tool 41 s over 22 calls, orchestration gap 5 s. (a) Does the invariant close, and if not, name three plausible homes for the remainder. (b) You may halve exactly one of model time, tool time, or the gap. Compute each end-to-end saving. (c) Ten of the 22 tool calls are independent of one another and average 1.9 s; what does issuing them concurrently buy, and what does it not? Solution sketch: (a) 26 + 41 + 5 = 72 against 84, so 12 s is unattributed — candidates are provider-side queueing counted in neither phase, retries logged under the wrong id, and serialization of the transcript outside the measured gap. Fix the accounting before optimizing anything. (b) Model → 13 s saves 13 s (15%); tool → 20.5 s saves 20.5 s (24%); gap → 2.5 s saves 2.5 s (3%). (c) Ten calls at 1.9 s serially is 19 s; run concurrently they cost about one round, ≈1.9 s, saving ≈17 s — comparable to halving all tool time, for no change to any tool. It buys nothing on the dependent calls, and it makes the failure analysis harder because the calls no longer have a total order in the transcript.
4. The gameable verifier. You add a benchmark as an in-loop check and the agent reports a 6× speedup on the second step. (a) List three mechanisms by which that number could be real to the verifier and false to you. (b) Design the cheapest change to the verifier that eliminates each. (c) Explain why a held-out check run by you is still required after all three fixes. Solution sketch: (a) The benchmark read a cached result; the change altered the workload rather than the system (fewer tokens generated, shorter input); the invariant was silently violated, so the fast path is computing something else. (b) Force a cold run and include the input hash in the reported record; assert the workload descriptor matches the frozen one; assert byte-for-byte output equality at temperature 0 and fail the check if it does not hold. (c) Because each fix closes a mechanism you thought of, and the agent optimizes against the check as it stands — the only evidence that the win generalizes is a check it was never able to optimize against.
5. Freeze what, exactly. Assignment 4 will re-measure your agent after replacing the frontier API with a model you serve. (a) List everything that must stay identical between the two measurements for the comparison to mean anything. (b) For each item, name the way it drifts by accident rather than by decision. (c) One thing on your list cannot be held fixed. Identify it and say what you report instead. Solution sketch: (a) The task set, the success criterion, the scoring script, the metric definitions and their price basis, the harness and its warmup/repeat counts, the temperature and sampling parameters, and the tool implementations. (b) A task reworded to be "clearer"; a criterion loosened because a failure was annoying; a scorer patched to handle a new output format; a price sheet updated silently; a tool improved mid-stream. (c) The model, which is the entire point of the assignment — so it is reported rather than held: name and version, date, decoding parameters, and the serving configuration, so that the difference you measure has exactly one cause.

Reading guide

SWE-agent — required. Read it as a design paper, which is what it is. The contribution is not a model and not a scaffolding trick: it is the observation that the interface an agent acts through — what the tools are, what they return, how errors are phrased, what gets truncated — determines what the agent can accomplish, and that the same model behind a better interface is a different system. Pay attention to the specific interface decisions and to how they were evaluated, because the evaluation is §5.3's problem solved at paper scale. Hold this question while reading: which of their interface decisions is a decision you have already made by accident in your own agent?

Reflexion — optional. §5.2 with the loop closed automatically: the verifier's feedback is written in language and re-enters the context, so the agent's next attempt is conditioned on its own critique. Read it for the mechanism, then price it with §5.2's arithmetic — the critique is v tokens per attempt riding in every later prompt, and the paper's trajectories are long.

AutoGen — optional. The vocabulary for §5.6: conversation between agents as the programming model. Read it for the composition patterns and for what it does not give you, which is any account of what the arrangement costs — that is §5.6's crossover arithmetic, and the serving-side version arrives on Nov 23.

Looking ahead

Part I ends here, and it ends by turning the camera around. You have designed an agent: a loop, a context policy, a tool interface, a verifier, and a harness that measures all of them. Now look at the request stream that design emits, because Part II is fifteen meetings on the machine that has to serve it.

That stream has a shape, and it is nothing like the chat traffic serving systems were built for. Prefixes are enormous and shared, within a session and across the branches of a fan-out — the 20-step task submits 97,000 prompt tokens of which only 7,700 are distinct, so 92% of what you send is something the system has already seen, and the fraction grows with the square of the step count. Decodes are short and numerous: a step emits a tool call, not an essay, and almost all of the compute in an agent workload is prefill. Requests are strongly dependent — the next one cannot even be formed until the previous returns. And sessions stall: a 16,384-token session on a 13B-class model without grouped-query attention holds about 13.4 GB of KV cache while a tool runs and nothing at all is generated, so roughly six such sessions fill an 80 GB card while producing zero tokens.

Every one of those properties is a property of the program you just wrote, and the request API throws all of it away — the serving system receives finished strings and cannot tell that request 7 continues request 6, that 8 through 12 are branches of one fan-out, or that only the last one is latency-sensitive. That gap is the argument of Lecture 4's required reading, and it is the reason Part II exists.

Part II opens on Sep 28 with LLM serving basics, then GPU architecture and kernels (Sep 30, Oct 5), batching and scheduling (Oct 7, Oct 14), routing (Oct 26), KV-cache optimization (Oct 28), the prefix cache that the 92% figure is about (Nov 4), quantization (Nov 9), speculative decoding (Nov 11), and then two meetings on agent serving specifically — Nov 18 on declared structure, the request DAG, and scheduling agentic programs, and Nov 23 on tool stalls, session state, and multi-agent workloads. You will meet today's design decisions there from the other side, as things a scheduler wishes it could see.