Three classes have described agents: what one is and what a session costs, how to write the loop, and how to hold that loop to a specification you can check. None of them answered the question you will actually face for the rest of the semester, which is how to get useful work out of one. That is this note's subject, and it is a systems question rather than a matter of taste, because the quantity that decides whether delegation pays is not the model's quality but the cost of checking its output. By the end you should be able to compute whether a task is worth handing to an agent, design the verifier that makes the handoff work, price the four cost knobs you hold as a user, and say when fanning a task across several agents is cheaper than running it in one context — the workflow that Assignments 4 and 5 are graded on.

Where this sits

Lecture 3 priced a session from the outside. Lecture 4 made the loop's four decisions yours. Lecture 5 made the loop accountable — the specification it is held to, the verifier inside it, and the frozen task set that says whether a change helped. All three put you at the keyboard of the agent. Today you are inside it: you are the outer control loop, the one that specifies the task, reads the result, and decides whether to accept it, and every quantity in this lecture is a property of that outer loop rather than of the model. It is also the last class before we go under the API, and several of today's numbers — the cache-hit arithmetic in §6.5, the duplicated preambles in §6.6 — turn out to be properties of that machine rather than of your design, which is the reason Part II exists.

Instructor notes — Timing plan

75-minute plan (Mon/Wed 11:15am–12:30pm, SEC LL2.221), kept in case this material is ever given the reserved Oct 19 slot. It is not currently lectured.

Time	Segment	Notes
0–4	Framing	"You have been the thing outside the loop for three classes. Today you are in it." No assignment goes out today; say so early so nobody waits for it.
4–14	§6.1 The delegation arithmetic	Derive p_min live. This is the spine of the lecture.
14–24	§6.2 The prompt is a specification	Before/after, then the 19× late-discovery number.
24–36	§6.3 Give the agent the verifier	The 0.36 → 0.82 lift. Protect these twelve minutes.
36–41	§6.4 Read the run, not the answer	Fast — it is mostly Lecture 5's counters pointed at yourself.
41–51	§6.5 The four cost knobs	Build the three-row prompt-ordering table live; the 5.8× is the memorable one.
51–70	§6.6 Multi-agent, and optimization work	The second centerpiece. Derive the k ≈ 6 crossover, then walk the optimization workflow.
70–73	§6.7 When not to use an agent	Read the table off §6.1; do not re-derive.
73–75	§6.8 Where this lands	Assignment dates, the bonus, one sentence on what comes next.

If running long: compress §6.4 to the three counters and cut §6.6's pattern catalogue to orchestrator/worker plus judge panel. Never cut §6.1 or the §6.6 crossover derivation — the first is the lecture's thesis and the second is the only quantitative reason anyone should believe a multi-agent design.

Learning objectives

By the end of this class you should be able to:

1. Compute the break-even success probability at which delegating a task to an agent costs you less time than doing it yourself, and say which term in that expression dominates.
2. Rewrite an underspecified request as a checkable specification, and price what discovering a specification error late in a session costs relative to discovering it early.
3. Compute the effect of an in-loop verifier on task success across a multi-step task, and choose a verifier from a cost-strength ladder.
4. Name the four cost knobs a user holds, and compute what prompt ordering is worth on a session whose re-sent fraction you know.
5. Derive the number of independent work items above which fanning out across sub-agents submits fewer tokens than running the work in one context, and say what the duplicated preambles cost.
6. Lay out a multi-agent workflow for a performance-optimization task, including the ceiling calculation that decides where to spend the agents and the measurement discipline that decides which candidate won.

6.1 The arithmetic of handing work over

Start with the decision itself, because it has an answer. A task takes you M minutes to do yourself. Delegated, it costs you S minutes to specify, and then V minutes to check each attempt the agent produces — every attempt, including the failures, because you cannot tell a failure from a success without checking. If each attempt succeeds independently with probability p, the expected number of attempts until one succeeds is 1/p, so your expected time is:

Notice what is absent. The agent's own wall-clock — the minutes it spends thinking and running tools — does not appear, because it is not your time. That is the entire economic case for agents, and it is also the most common way people throw the gain away: watching the run costs you the wall-clock you were trying not to spend. Notice also what is present. V is divided by p, so verification cost is multiplied by unreliability, and the product is the thing that decides.

Which of the two levers is worth your engineering effort? Differentiate, or just evaluate. On the first row, halving V from 5 to 2.5 minutes saves 4.2 minutes; raising p from 0.6 to 0.8 saves 8.33 − 6.25 = 2.1 minutes. A verifier twice as cheap beats a model a third more reliable, and you can build the first one this afternoon while the second is somebody else's release schedule. This is the practical form of the observation Lecture 3 made about the workload taxonomy: coding is the workload agents are good at because compilers and test suites drive V toward zero, not because the models were trained on more of it.

The same expression prices the money as well as the time. If an attempt costs c dollars, expected spend per completed task is c/p — Lecture 3's cost-per-completed-task, with attempts averaging ≈$0.29 (successes ≈$0.18, failures running to the ≈$0.55 budget) and 70% success giving the $0.42 it computed there. Both denominators are p, which is why §6.3 is about raising it and §6.5 about lowering c.

Instructor notes

Minutes: 10. Board: Write S + V/p < M and box it, then derive p_min = V/(M − S) beside it. Then the two worked rows as two columns, changing only V and p, and circle V. Ask the room: "You have a task you would rather not do. What do you need to know before handing it to an agent?" Collect answers; most will say something about the model. Steer to: how long it takes to check the answer. Expect confusion: Students read V as a small constant. Push: how long does it take to verify a market analysis, a schema migration, a proof? For some tasks V ≈ M, and then delegation can never pay. Common wrong answer: "The agent is fast, so it wins." The agent's wall-clock is not in the expression; your attention is. If you sit and watch, you have put it back in. If short on time: The formula and the second worked row. The sensitivity comparison can be read.

6.2 The prompt is a specification, and it is read once by a stranger

Lecture 4 made the case for tool schemas: your caller reads the documentation fresh at every call, and no compiler checks it. The task description is the same object one level up. It is a specification handed to a competent stranger who will not ask you a clarifying question, and Lecture 3 gave the failure class it produces when it is thin — specification failure, the one where every component behaved and the words underdetermined the intent.

A request as it is usually typed:

Make the serving script faster.

The same request written so that a stranger could satisfy it and you could check that they had:

Goal: reduce median time-per-output-token on bench/trace_200.jsonl by at least 20%.
Measure with: scripts/bench.py --trace bench/trace_200.jsonl --repeat 5 --warmup 1
  (report median and min/max across the five runs; the harness prints all three)
Invariant: generated text must match the current output byte-for-byte at temperature 0.
Out of scope: changing the model, the tokenizer, or the sampling parameters.
Budget: stop after 40 tool calls or 20 minutes and report what you have.
Done when: the measured median improves by >= 20% with the invariant holding, or you
  can show why it cannot, with the profile that says so.

Six things changed, and each removes a way for the run to end in an argument: a metric replaces an adjective, a measurement procedure replaces "faster", an invariant states what must not change, an out-of-scope list forecloses the cheap wrong answers, a budget bounds the run (Lecture 4's safety property, stated where the agent can see it), and a done-condition tells the loop when to stop rather than leaving is_final to guess.

Front-loading the specification is not tidiness; it is the cheapest point on a curve. Take Lecture 4's session shape — a 2,000-token pinned preamble and 300 tokens appended per turn — and write C(k) for the cumulative prompt tokens submitted through turn k:

And the tokens are the cheap part. A specification error discovered at turn 18 has also spent your V, possibly written to the world through a non-idempotent tool (Lecture 4's §4.5), and — worst — may not be discovered at all, because a fluent answer to the wrong question does not announce itself.

Instructor notes

Minutes: 10. Board: Put the one-line prompt up. Ask for fixes and write them as a list; then reveal the six-line version and check off which of their suggestions it contains. Then C(18) vs C(2) in the corner: 81,900 / 4,300 = 19×. Ask the room: "Which line of the good version would you drop if you were in a hurry?" Whatever they pick, ask what failure that line was preventing. The done-condition is the one they drop most and the one that costs most. Expect confusion: "The agent can ask me." It can, but each question is a round trip you pay for in V, and models under-ask by default. Common wrong answer: "Longer prompts are better prompts." No — checkable prompts are better prompts. A page of vague context is worse than five specific lines, and it rides in every subsequent call. If short on time: Keep the metric/invariant/done-condition trio and the 19×.

6.3 Give the agent the verifier

Lecture 3 established the compounding problem: at 0.95 success per step, a 20-step task completes 0.95²⁰ ≈ 36% of the time, and no amount of enthusiasm about per-step accuracy repairs that. The fix is not a better model. It is a verifier inside the loop, so that a failed step is caught and retried while its error is still local, before it poisons the context for the eighteen steps that follow.

Model it. Let q be the per-step success rate without a verifier and c the fraction of step failures the verifier catches and the agent then repairs. The effective per-step rate is:

Verifiers form a ladder, and the discipline is to use the cheapest rung that catches the errors you actually make. A type or syntax check costs milliseconds and catches a narrow but very common class. Unit tests cost seconds. Integration tests and a full build cost minutes. A benchmark costs minutes and returns a noisy number, which §6.6 treats as its own problem. You cost minutes to hours, and you are the rung that catches everything the others cannot express. Climb only as far as the error class demands.

Three properties decide whether a verifier helps or merely runs. It must be legible: Lecture 4's "errors are data" applies here, because a verifier that fails with an empty diff or a stack trace tells the model nothing it can act on, and a failing check the model cannot read is a step that will fail again identically. It must be bounded: the verifier's output enters the transcript and is re-sent on every remaining call, so a test runner that dumps 20,000 tokens of output at step 4 of a 20-step task costs the 320,000 prompt tokens Lecture 4 priced — cap it in the tool, at the point of production. And it must be non-gameable, which is the uncomfortable one. The agent optimizes what you measure. A test that asserts nothing passes; a benchmark that reads a cached result gets faster without the code changing; a lint rule satisfied by a suppression comment is satisfied. Every weak verifier converts a visible failure into a confident one, which is strictly worse than having no verifier at all, and the only general defence is a check the agent was not optimizing against — held out, run last, by you. The held-out task sets the later assignments score against are built on exactly that principle.

Instructor notes

Minutes: 12. Centerpiece — protect it. Board: q' = 1 − (1−q)(1−c) at the top. Then 0.95²⁰ = 0.36 and 0.99²⁰ = 0.82 underneath, with "+25% wall-clock" beside them. Then the ladder as five rungs with their costs, vertical. Ask the room: "Your agent's task has 20 steps and each is 95% right. Would you rather have a model at 99% per step, or a two-second test suite the agent can run?" They are the same 0.82; one you can build today. Expect confusion: Verification is thought of as something done at the end, by the user. The whole point is that it belongs inside the loop, where a failure is still cheap and local. Common wrong answer: "Add more tests." Ask what the tests assert. Then tell the cached-benchmark story from §6.6 and let it land — a verifier the agent can satisfy without doing the work is the failure mode of this entire section. If short on time: Keep the formula, the 0.36 → 0.82 line, and the non-gameable paragraph.

6.4 Read the run, not the answer

When a run goes wrong, the answer is the least informative artifact it produced. Lecture 3 gave the diagnostic order — tool, then harness, then specification, then model — and the reason for that order is that the first three leave evidence and the fourth is the explanation that requires no work. As a user your job is to keep the evidence.

Three measurements are worth taking on your own runs, and Lecture 5 specified them precisely enough to implement. Tokens by component: preamble, tool results, model replies, and re-sends, each as a share of input, because the component that dominates is the one worth attacking and it is almost never the one you assumed. Latency by phase: model time, tool time, and the orchestration gap, which must sum to wall-clock — anything unaccounted for is cost you cannot optimize. Model calls per completed task, reported as a distribution rather than a mean, because the tail is where the money goes.

And judge on a set, not an anecdote. Lecture 3's arithmetic is worth restating because it is so easily forgotten in a demo: ten clean runs are consistent, at 95% confidence, with a true failure rate of 26%. Five to ten fixed tasks, several runs each, median and spread reported, cost per completed task rather than per run. That is a morning's work and it is the difference between knowing your p and guessing it — and §6.1 says every decision you make about delegation depends on p.

Instructor notes

Minutes: 5. Deliberately brisk; this is Lecture 5's §5.4 pointed at the student rather than at the system. Board: Three lines only: tokens by component, latency by phase, calls per completed task. Then "10 clean runs ⇒ true failure rate could be 26%". Ask the room: "How many of you know your agent's success rate to within ten points?" The silence is the point. If short on time: The three counters and the 26%.

6.5 The four cost knobs you hold from outside

You do not control the engine. Part II is about the people who do. From outside the API you nevertheless hold four knobs, and they are worth more than their reputation.

Model choice per subtask. Nothing requires one task to use one model. Mechanical steps — reformatting, extracting a field, deciding which of three tools to call next — are frequently satisfied by a much cheaper model, and the expensive one is needed for the two or three steps that carry the judgement. The decision returns on Oct 26 as routing, when it becomes a scheduler's problem instead of yours.

Output caps, in the tool. Lecture 4's arithmetic: one uncapped 20,000-token tool result at step 4 of a 20-step task rides in sixteen later prompts and costs 320,000 prompt tokens against the task's own 97,000; capping the tool at 2,000 tokens brings the total from 417,000 to 129,000, a 3.2× reduction from one line in the tool.

Context policy. Lecture 4 priced three: 97,000 tokens for keep-until-full, 83,500 for a pinned sliding window, 75,100 for summarize-and-compact — and made the case that the ≈23% spread is the wrong reason to choose. Choose on the failure column; take the tokens as a bonus.

Prompt ordering. This is the knob nobody thinks of as a knob, and it is the largest. Lecture 5 counted the re-sends: of the 97,000 prompt tokens a 20-turn session submits, only 7,700 are distinct — the last call's prompt — and 89,300 are exact re-sends of a prefix the provider has already read. Serving systems exploit that with a prefix cache (Nov 4; caching your own repeated calls is one of the directions the final project lists). Whether your session benefits is decided by where you put the tokens that change.

Write f for the fraction of the fresh-input price at which re-read prefix tokens are billed, and note that the reusable prefix ends at the first token that differs between two calls. Everything after that first difference is fresh, however stable it looks.

The practical rules follow from the table. Put the stable material first and never insert into it — append. Do not stamp the time, a request id, or a random seed into the system prompt. Serialize tool schemas in a fixed order (a set's iteration order is not fixed). If a session needs the current time, let the agent call a now() tool, whose result lands in the newest turn where it belongs. Two honest caveats: f, and whether cached tokens are billed at all, are provider-specific and change; and reuse also depends on the server keeping your prefix resident, which is an eviction decision made by somebody else — the subject of Nov 4.

Instructor notes

Minutes: 10. Board: The four knobs as four words. Then build the three-row table live, deriving 59,019 in front of them — the middle row is the one that surprises, and the surprise is "reuse stops at the first difference". Ask the room: "Your system prompt starts with the current date. What did that cost?" Let them find 5.8× themselves from the numbers on the board. Expect confusion: Students think the cache matches anywhere in the prompt, like a diff. It matches a prefix. Say it twice. Common wrong answer: "Just make the context smaller." That is a different knob and a smaller one; the ordering knob costs nothing and loses nothing. If short on time: Skip model-per-subtask and the caps (both are recaps) and spend the whole slot on the table.

6.6 More than one agent

Lecture 4 introduced sub-agents and named their real payoff: context isolation. A sub-agent reads forty files and returns three lines, and the thirty-seven irrelevant results never enter the parent's transcript. Today we price that, add the patterns worth knowing, and then apply the whole apparatus to the task the rest of this course is about — making something faster.

There are exactly three good reasons to use more than one agent. Context isolation, as above: the parent's working set stays small, and Lecture 4's §4.3 explains why that is worth paying for. Independent perspectives: two agents that share no context can disagree, and disagreement is a signal you cannot get from one agent asked twice. Wall-clock, when the work is genuinely independent. And one bad reason, which is the common one: hoping that more agents will fix a result that is wrong because the specification was thin or the verifier was weak. Fanning out a bad specification buys k confidently wrong answers.

What a fan-out costs

Compare two designs on k independent work items — files to summarize, candidate patches to try, sources to check — with Lecture 4's shape: a preamble of P = 2,000 tokens, g = 300 tokens appended per item, and, in the fan-out, a return summary of r = 100 tokens per worker.

Now notice what those 40,000 duplicated tokens are: k prompts sharing an identical 2,000-token prefix, which is precisely the pattern §6.5 just priced. Count the fan-out's distinct tokens — one worker preamble, twenty items, one parent preamble, twenty summaries — and you get 12,000, so 38,000 of the 50,000 are cache-eligible and the bill at f = 0.1 is 12,000 + 3,800 = 15,800 units, another 3.2× below the uncached figure. Which yields the sentence worth carrying into Part II: the economics of a multi-agent design are a property of the serving system underneath it, not of your design. The same fan-out is an extravagance on a system with no prefix cache and nearly free on one that reuses shared prefixes across concurrent requests. Part II is the other side of that difference, and Nov 23 is its exact counterpart to this section: the same multi-agent workload, seen by the scheduler that has to run it.

Four patterns, and how each fails

Orchestrator and workers — one agent decomposes, k workers execute one item each, the orchestrator integrates. The right shape for the independent-item case above. It fails at integration: workers make incompatible assumptions, and k workers with side-effecting tools multiply Lecture 4's idempotency problem by k, since two workers can retry the same non-idempotent write.

Pipeline without barriers — every item flows through all stages independently, so item A can be in stage 3 while item B is still in stage 1. Wall-clock is the slowest single item's chain, not the sum of the slowest stage times. It fails only when a stage genuinely needs all of the previous stage's results — deduplication across items, or an early exit — in which case you want a barrier and should say so.

Judge panel — N agents check the same artifact independently and you take the majority. If each catches an error with probability 0.7 and they are independent, all three miss with probability 0.3³ = 0.027, against 0.3 for one: 11× better. The failure mode is in the premise. Three copies of the same model given the same prompt share their blind spots, and correlated judges collapse the 0.027 back toward 0.3. The mechanism is diversity — different lenses, different evidence, ideally different verifiers — not redundancy.

Critic loop — one agent produces, another criticizes, repeat. Effective, and unbounded by construction: it needs Lecture 4's stopping condition, a cap on rounds, and a rule for when "no substantive objection" ends it.

Multi-agent for performance optimization

This is the workflow Assignment 5 asks for, so it is worth laying out as a procedure rather than a sentiment. The task: a system you own is too slow or too expensive, and you have agents to spend on it.

Step 0 — compute the ceiling before you spawn anything. Lecture 5 said it: profile first, and the profile tells you what the best possible outcome is. If wall-clock is 60% tool wait and 40% model time, then driving model time to zero yields 1/0.6 = 1.67× end to end, and eight agents attacking the model path cannot beat it. If the session submits 97,000 prompt tokens of which 7,700 are distinct, prefix reuse cannot exceed 12.6×. Spend agents in proportion to the ceiling, not to your interest in the component. Amdahl's law does not care how much you enjoy writing kernels.

Step 1 — one shared profile. Produce it once, and hold it to Lecture 5's closing invariant: model time plus tool time plus the orchestration gap equals wall-clock, and the token components sum to input. An unattributed component is a component nobody will optimize.

Step 2 — fan out candidates, in isolated workspaces. One agent per candidate optimization, each with its own copy of the tree — a worktree, a container, a directory. This is a correctness requirement rather than hygiene: k agents editing one checkout is a lost-update problem, and you will spend more time untangling it than the fan-out saved. Six candidates on a 1.67× ceiling is a reasonable fan; twenty is theatre.

Step 3 — one measurement harness, and treat noise as noise. Every candidate is measured by the same script, on the same inputs, with the same warmup, and the metric is a median over repeats with the spread reported. The discipline that matters: a candidate that wins by less than run-to-run variation did not win. If run-to-run σ is 4% of the metric and you want to resolve a 3% improvement at roughly two standard errors, you need 2 · 4/√n ≤ 3, so n ≥ 7.1 — eight runs. A 20% improvement needs one. Measure in proportion to the size of the claim, and always measure a no-op candidate: a change that does nothing should come back at 1.00×, and if it comes back at 8× your harness is caching something.

Step 4 — select, then integrate one at a time. A judge step (you, or an agent reading the measurement tables, with the tables as evidence rather than the candidates' own reports) picks the winners. Then merge them sequentially, re-measuring after each, because optimizations interact: two changes that each remove 20% of the same bottleneck do not compose to 40%.

Step 5 — check on something the candidates never saw. The agents optimized what you measured, so the benchmark is now the thing they are good at. A held-out workload is the only evidence that the win generalizes, and the assignments turn that principle into an assessment: your improvements are scored on held-out tasks they were never tuned against.

Instructor notes

Minutes: 19. The second centerpiece. Board: C_single and C_fan side by side, then 97,000 vs 50,000, then "40,000 of the 50,000 is the same 2,000 tokens twenty times" and circle it — that circle is the handoff to Part II. Then the crossover quadratic and k ≈ 6. Keep the five optimization steps as five numbered lines; do not elaborate them on the board. Ask the room: "Twenty files to summarize: one agent or twenty?" Take a vote before the arithmetic, then run it. Then ask "three files?" and let them discover that the answer inverts. Expect confusion: Multi-agent is believed to be strictly more expensive, or strictly better. It is neither: there is a crossover, it is computable, and it depends on the serving system's prefix cache. Common wrong answer: "Use more agents to get a better answer." Only if they disagree independently; three copies of the same prompt share their blind spots. Use the 0.027-versus-0.3 number and then take it away by correlating the judges. If short on time: Cut the pattern catalogue to orchestrator/worker and judge panel, and state the payback arithmetic without deriving it. The crossover and Step 0 must survive.

6.7 When not to use an agent

§6.1 is a decision procedure, so it also tells you when to decline. The table is the lecture's negative image, and none of its rows is about the model being bad.

Signal	Why it is disqualifying	What to do instead
V approaches M — checking costs about what doing costs	p_min = V/(M − S) approaches 1, so no attainable reliability makes it pay	Build the verifier first, then reconsider; the verifier is the reusable asset
p below V/(M − S) on your measured task set	The arithmetic says you are paying to supervise	Narrow the task until p rises, or split it so each piece is checkable
M is small — a two-minute edit	S alone can exceed M	Do it
A tool with side effects and no request key	A timeout is three worlds (Lecture 4, §4.5), and a retry may duplicate	Add idempotency at the interface, or keep the write for yourself
Judging the output needs the expertise you were trying to skip	The verification is the work	Use the agent for the parts you can check, and do the judgement
The value is in the process — a design you must defend, material you are learning	Delegation removes the thing you wanted	Delegate the mechanical half only
Data that must not leave your machine	Not a cost question at all	A local model, or no agent

Two of those rows are the honest reason this course asks you to build agents rather than only use them: the second and the fourth are fixable by engineering, and knowing which failures are fixable is most of the skill.

Instructor notes

Minutes: 3. Board: Nothing new — point at p_min = V/(M − S) from §6.1 and read three rows off it. Ask the room: Nothing. Say plainly that "an agent could do this" and "delegating this pays" are different claims, and that the second one has arithmetic.

6.8 Where this lands on the course

Assignment 1 is due Sun Sep 20, 11:59pm, and this note is the practical half of it. Run your task on a small fixed set rather than once (§6.4), and if you have not yet built the cheapest verifier your task admits, that is the highest-value hour you can spend before the deadline — and you present what you built at Student sharing I on Sep 21, the morning after the deadline, where §6.4's four numbers are the difference between a report and a demo. Assignment 2 follows, out Sep 21 and due Sun Oct 4, and §6.3's "cap the verifier's output in the tool" is a line of your own code by then.

Further out, the two halves of this lecture split across the two later assignments. Assignment 4 (out Oct 26, due Tues Nov 10) replaces a frontier API with a model you serve yourself and asks you to recover quality — §6.2 and §6.3 are how you find out whether you have. Assignment 3 (out Oct 5, due Sun Oct 25) and Assignment 5 (out Nov 11, due Wed Dec 2) both hold quality fixed and drive cost down, the first from outside the API and the second with the serving stack in scope, and §6.6 is its procedure: ceiling first, fan out candidates in isolated workspaces, one harness, integrate one at a time, check on something held out. Assignment 5 is that procedure run for real on your own stack, which is why Step 5 is not optional advice. And §6.1's arithmetic is what decides how you spend your own semester — on the assignments, and on the ungraded optional project if you take one on.

Two policy notes. Agent use is expected in this course under two conditions, both on the policy page: disclose what you used, and take responsibility for what it produced. Separately, the bonus pays 2% for each well-documented problem current systems cannot solve. §6.1 is where those live: a task with a genuinely irreducible V, documented with the transcripts showing what checking it actually took, is worth more to this course than another task that worked.

Instructor notes

Minutes: 2. Board: Five handoffs, each one assignment due on a Sunday and the next out the Monday after: Sep 20 → 21 (A0 due; A1 out), Oct 4 → 5 (A1 due; A3 out), Oct 25 → 19 (A3 due; A4 out), Nov 10 → 2 (A4 due; A5 out), Dec 2 (A5 due). Ask the room: Nothing. Say that nothing goes out today, and that the highest-value hour before Sep 20 is spent building a verifier, not writing a better prompt.

Key takeaways

Numbers worth memorizing

Quantity	Value	Where it came from
Break-even success probability for delegation	p_min = V/(M − S)	§6.1
Your time, M = 60, S = 10, V = 5, p = 0.6	18.3 min (3.3× saving)	§6.1
Same task, V = 25, p = 0.4	72.5 min — worse than doing it	§6.1
Verifier lift, q = 0.95 and c = 0.8, over 20 steps	0.36 → 0.82 (2.3×) for +25% wall-clock	§6.3
Specification error found at turn 18 versus turn 2	81,900 versus 4,300 tokens (19×)	§6.2
Prompt ordering, 20-turn session, cached tokens at f = 0.1	97,000 → 16,630 billed units (5.8×)	§6.5
Same session, volatile token at the end of the preamble	62,817 billed units (only 1.5×)	§6.5
One context versus fan-out, 20 independent items	97,000 versus 50,000 prompt tokens (1.94×)	§6.6
Fan-out crossover, P = 2,000, g = 300, r = 100	k ≈ 6 items	§6.6
Duplicated preamble share of a 20-way fan-out	40,000 / 50,000 = 80%	§6.6
Three independent judges at 0.7 detection	miss rate 0.027 versus 0.3 (11×)	§6.6
Repeats needed to resolve a 3% win at σ = 4%	8 runs	§6.6
Ceiling from a 60% tool / 40% model split	1.67× end to end	§6.6

Self-check

Exercises

1. Delegate, then improve the verifier. A task takes you 120 minutes. Specification costs 20, checking an attempt costs 30, and your measured success rate is p = 0.45 over a 12-step run. (a) Does delegation pay, and what is the break-even p? (b) You add an in-loop verifier catching 60% of step failures. Recompute p and your time. (c) The verifier also means you spend 15 minutes checking rather than 30. Recompute. (d) Which of the two effects in (b) and (c) mattered more, and what does that say about where to put engineering effort? Solution sketch: (a) 20 + 30/0.45 = 86.7 min against 120 — it pays, 1.38×; p_min = 30/100 = 0.30. (b) Per-step q = 0.45^(1/12) = 0.9356, so q' = 1 − 0.0644 · 0.4 = 0.9743 and p' = 0.9743¹² = 0.731; time = 20 + 30/0.731 = 61.0 min (1.97×). (c) 20 + 15/0.731 = 40.5 min (2.96×). (d) The V reduction: it saves 33.3 minutes on its own (20 + 15/0.45 = 53.3 min) against the reliability gain's 25.7, but the two compound and both come from the same artifact — one verifier raised p and lowered V, which is why §6.1 calls it the reusable asset.
2. Prompt ordering on a different shape. A session has a 3,000-token preamble, appends 400 tokens per turn, and runs 15 turns; cached prefix tokens bill at f = 0.2. Compute cumulative prompt tokens, then the billed units for the three placements of a volatile token in §6.5's table. Solution sketch: Cumulative = 15 · 3,000 + 400 · 105 = 87,000; distinct = 3,000 + 400 · 14 = 8,600, so 78,400 are re-sends. Volatile only in the newest turn: 8,600 + 0.2 · 78,400 = 24,280 units (3.58×). Volatile first: no reuse, 87,000 (1.0×). Volatile as the preamble's last token: consecutive calls share 2,999 tokens, so cached = 14 · 2,999 = 41,986, fresh = 45,014, billed = 45,014 + 8,397 = 53,411 (1.63×). The middle placement recovers about a fifth of the available saving.
3. When the cheapest layout is illegal. Twelve work items, P = 2,000, g = 300, r = 100 — but item i needs item i−1's result. (a) Cost in one context. (b) Cost of a full 12-way fan-out, and why the number is irrelevant. (c) Cost of three parallel groups of four, sequential within a group, with a parent reading three summaries. (d) State the general rule this exercise is an instance of. Solution sketch: (a) 12 · 2,000 + 300 · 66 = 43,800. (b) 12 · 2,300 + 2,000 + 1,200 = 30,800 — cheaper, and unavailable: the dependency means worker i cannot start before i−1 finishes, so the layout does not exist. (c) Each group 4 · 2,000 + 300 · 6 = 9,800, three groups 29,400, parent 2,000 + 300 = 2,300 → 31,700, a 1.38× saving with 3× the parallelism. (d) Fan out along the independent axis only; a dependency chain is a pipeline, and the cheapest-looking layout is often the one the dependency structure forbids.
4. Allocating agents against a profile. A profile attributes wall-clock as 55% tool wait, 30% model time, 15% orchestration. (a) Give the end-to-end ceiling for driving each component to zero. (b) You have eight candidate-exploration agents. Allocate them and justify it with (a). (c) Suppose the winners cut tool wait by 60% and model time by half — what is the measured end-to-end speedup? (d) If the workload costs $400/month and the fan-out cost $4 of agent spend, what is the payback period? Solution sketch: (a) Tool 1/0.45 = 2.22×; model 1/0.70 = 1.43×; orchestration 1/0.85 = 1.18×. (b) At least half the agents on the tool path — it is the only component whose ceiling exceeds 2× — perhaps five on tool wait, two on the model path, one on orchestration; the allocation follows the ceilings, not the interest. (c) Components become 0.22 + 0.15 + 0.15 = 0.52, so 1.92×. (d) New cost 400/1.92 = $208, saving $192/month ≈ $6.40/day: payback under a day. The $4 was never the interesting number; the 2.22× ceiling was.
5. The harness that lies. Your benchmark script caches results between invocations. The first run of the session takes 100 s and every later run of anything takes 12 s. Six candidate agents each measure themselves after a baseline run. (a) What speedup does each report? (b) What does a no-op candidate report? (c) Name the two harness changes that would have caught this, and (d) connect it to §6.3's third property of a verifier. Solution sketch: (a) 100/12 = 8.3×, all six, suspiciously alike. (b) 8.3× — which is the tell, and the reason a no-op control belongs in every candidate set. (c) Include a no-op candidate, and clear the cache (or randomize run order and re-measure the baseline last) so that no measurement depends on its position in the sequence. (d) It is verifier gaming without any agent misbehaving: the agents optimized the measurement, exactly as instructed, and a metric that can be satisfied without doing the work will be. A held-out workload would have shown 1.0×.

Reading guide

SWE-agent — required. Read the introduction and the section that defines the agent-computer interface, then the ablations that vary the interface while holding the model fixed. That is the paper's argument and it is today's lecture with measurements attached: the interface — how a file is shown, how an edit is applied, whether a failed edit reports why — moves the success rate as much as a model change would. Skim the benchmark tables. Hold this question while reading: which of your tools has an interface designed for a human reader rather than for the model that actually calls it, and what would you change first?

Reflexion — optional. §6.3 with the loop closed automatically: the verifier's complaint is written in language and re-enters the context as the next attempt's input. Read the method, skip the benchmark sweep. Hold this: what supplies the signal, and what happens to the loop on a task where no cheap signal exists?

AutoGen — optional. Read it for the pattern vocabulary of §6.6 — who talks to whom, and who decides when a conversation ends. Then price one of its examples with §6.6's two expressions and ask what the duplicated preambles cost, a question the paper does not need to ask and a serving system very much does.

KernelBench — optional, and the closest thing to Assignment 5 in the literature. Read how correctness and speed are both defined mechanically, which is what makes the task automatable at all. Hold this: the benchmark is the verifier, so what would a system that games it look like, and which of §6.6's five steps is the defence?

Looking ahead

Part II opens on Wed Sep 28 with LLM serving basics, and it begins where §6.5 and §6.6 stopped: you now know that a session re-sends 92% of its prompt tokens and that a fan-out sends the same preamble k times, and the question of who is allowed to not recompute all of that is the engine's, not yours. The prefix cache that makes both cheap is Nov 4, two days after the final project is announced; caching your own repeated calls is one of the directions it lists. §6.5's model-per-subtask knob comes back as routing on Oct 26. The fan-out shape itself — many dependent calls, shared prefixes, sessions idle while tools run — is the subject of the two agent-serving meetings, Nov 18 and Nov 23, and of Nov 23 in particular, which takes today's multi-agent workload and asks how a serving system should schedule it; by then you will have built the workload those papers are trying to serve. Between now and Sep 20, the most useful thing you can do for Assignment 1 is the cheapest thing in this lecture: write the verifier, run the task five times, and report the spread.