Lecture 23 closed on tasks that arrive with a spec and a verifier already attached — an issue plus hidden tests, a kernel that either compiles and matches a reference or does not — and on the discipline of never trusting a number a model reported about itself. Today's papers try to automate the thing that produces the specs: the research loop itself — hypothesize, experiment, analyze, iterate. Two claims to test against the readings. MLGym's honest headline is that frontier agents beat research baselines almost entirely by tuning hyperparameters and applying known recipes; AI Scientist-v2's is that an agentic tree search carried a machine-generated idea all the way through human peer review — with caveats that do most of the epistemic work. The systems angle closes the course's own loop: a research loop is an agent trajectory (Lecture 4's 20-step ledger with a GPU experiment where the tool call used to be), so it has a token bill you can compute in this course's own units, and every optimization in Part II directly multiplies how many hypotheses a fixed research budget buys. By the end you should be able to price an AI-Scientist-style run in tokens and dollars and say which term dominates, place a result on the reproduce → improve → discover ladder, compute what best-of-N selection does to an acceptance statistic, argue what evidence would show an agent did research, and know what is due when between now and the end of the semester.
Lecture 23 (LLM for systems research I) established the pattern for LLMs applied to systems work: the wins came where a task had a crisp objective and an external verifier; the failures came where the verifier could be optimized against. Lecture 23 owns that argument and the measurement discipline that follows from it — report the distribution and the budget, never the maximum alone — and this note assumes both rather than re-deriving either. Today is the same lesson at the largest scope, where the thing being verified is "research," and it is the end of the chain: Part I priced a model invocation (Lectures 1–2) and an agent (Lectures 3–5), Part II shrank every term of that price across its meetings from Sep 28 to Nov 23, and today those prices return as the budget line of an automated research program. There is no next topic lecture — student sharing IV (Nov 30 and Dec 2) closes the semester; §24.9 and Looking ahead are the exit map.
Standing assumptions, unchanged from Lecture 2 §2.13–§2.16 and used throughout Part II. Reference 7B: N = 6.74B, 13.5 GB of bf16 weights. One H100 SXM: 3,350 GB/s, 989 TFLOP/s dense BF16, prefill at 50% of peak = 494.5 TFLOP/s, and 62.5 GB of KV budget. The agent ledger is Lecture 4's 20-step designer — 97,000 submitted prompt tokens, 7,700 distinct, 92% re-sent, ≈80 output tokens per step. API rates are Lecture 3's illustrative $3 / $15 per Mtok (input / output), and cached input reads are billed at 0.1× the input rate — illustrative, like the rates themselves. GPU rent is an illustrative $3 per H100-hour. Where a number here disagrees with an older note, Lecture 2 is the one that is right.
75-minute class (Mon/Wed 11:15am–12:30pm, SEC LL2.221). Student-led discussion; the instructor takes back exactly one segment (§24.6) for board work.
| Time | Segment | Notes |
|---|---|---|
| 0–8 | Framing + calendar teaser | Task-vs-loop distinction on the board; one line per wrap deadline so nobody is surprised later. |
| 8–25 | MLGym deep-dive (presenters) | §24.2–24.3. Push presenters to the aggregation question and the ladder placement, not the leaderboard. |
| 25–42 | AI Scientist-v2 deep-dive (presenters) | §24.4–24.5. The three caveats must each get named; do the best-of-N arithmetic live if presenters skip it. |
| 42–62 | §24.6 Pricing the research loop | Centerpiece — instructor-led, never cut. $0.315 → $0.074 per node and 158-vs-675 attempts must be on the board when the bell goes. |
| 62–70 | Synthesis + discussion seeds | §24.8 in brief, then seeds 1–3. Seed 1 (an MLGym for serving) is the one to protect. |
| 70–75 | Course wrap logistics | §24.9. Fixed five minutes; announcement, not content — but it is the last topic class, so never cut it either. |
Reading-only, not scheduled: §24.7 (the serving system underneath a research agent) composes Lecture 2/3/5 material and adds no new mechanism; mine it for discussion seed 1 if a segment finishes early. Exercises 3 and 5 point back into it.
If running long: compress §24.8 to its three headings and one sentence each, and cut discussion seeds 4–5. Protect §24.6 in full and the five logistics minutes.
By the end of this class you should be able to:
N selection inflates an acceptance statistic, and state what must be reported alongside any "an agent's paper was accepted" claim.Lecture 23's papers had a fixed task and an external verifier: the tests pass or they do not, the benchmark number moves or it does not. Today's object is the loop that produces tasks: hypothesize → experiment → analyze → iterate. Map the four stages onto Lecture 4's agent loop and they land exactly — a hypothesis is a plan, an experiment is a tool call, analysis is observation processing, iteration is the context update that feeds the next step. The research loop is the agent loop with very expensive tools: where Lecture 4's designer called grep and paid milliseconds, this loop calls train.py and pays GPU-hours. That single substitution in the tool slot — milliseconds to GPU-hours — is the whole of §24.6 and most of §24.7.
The loop itself is old. George Box described scientific modeling as propose a model → fit and criticize → revise, decades before anyone put a language model near it, and the optional reading (Li et al., 2024) runs exactly Box's loop with an LLM in the propose-and-criticize seats — over probabilistic programs, a formal language in which a proposal can be mechanically fit and scored. That formality is worth noticing now, because it is the property today's two required papers give up: MLGym keeps a mechanical score but opens the task, and AI Scientist-v2 opens everything and scores with models.
Which is the structural difference from Lecture 23, and everything hard about today follows from it: there is no ground-truth verifier for research. A failing test is unambiguous. "This result is interesting, novel, and correct" is not — it is a judgment, and judgments can be produced by humans (slow, expensive, inconsistent) or by models (fast, cheap, and gameable by exactly the optimizer they are judging). MLGym's answer is to restrict research to tasks where a metric exists; AI Scientist-v2's answer is to accept model judgment inside the loop and human peer review at the end. Neither answer is complete, and §24.5 is about the gap.
The punchline to carry into both papers: automating the loop means automating the judge, and the judge is where both papers are weakest.
Minutes: inside the 0–8 framing. Board: Two columns — "Lecture 23: task + verifier" and "today: loop, verifier inside the loop." Under the second, the four stages mapped to plan / tool call / observation / context update. Ask the room: "What is the tool-call latency of this agent loop?" Answer: GPU-hours, not milliseconds — the number that drives §24.6. Expect confusion: Students hear "no ground-truth verifier for research" as "no evaluation is possible." Say: "There is evaluation. There is no cheap, executable, un-gameable evaluation — and Lecture 23 already showed you what an optimizer does to the gameable kind."
MLGym is a gym-style framework plus a benchmark, MLGym-Bench: 13 open-ended AI-research tasks spanning computer vision, NLP, reinforcement learning, and game theory. Each task ships a baseline implementation and a quantitative metric; the agent gets a sandboxed environment with shell access, the baseline code, GPUs, and a compute budget, and tries to push the metric past the baseline. The evaluated agents wrapped the frontier models of early 2025 — Claude-3.5-Sonnet, Llama-3.1-405B, GPT-4o, o1-preview, Gemini-1.5-Pro — and we will not relitigate which ranked where, because the ranking is the least durable part of the paper.
The durable part is the benchmark design, and the contrast with the closed verifier makes it sharp. Lecture 23 anatomized the closed kind — issue plus repository plus hidden tests, or a kernel that either compiles and matches a reference or does not — so take its properties as given here: a binary, executable oracle, under which "solved" is a fact. An MLGym task is open: a metric to push and no known optimum, so "solved" does not exist and the benchmark designer must invent what counts as "better." Best attempt or average attempt? Normalized by compute budget or not? Aggregated across 13 metrics with different units and different headroom how? These choices are not bookkeeping — they decide whether the leaderboard rewards search volume or search quality, which is precisely the distinction §24.3 turns on. The paper spends real effort here; check its evaluation-protocol section for exactly what is aggregated and how, because a best-attempt score and a mean-attempt score answer different questions — one measures what the agent can find, the other what it reliably does — and §24.6 shows that the first answer is purchasable with money.
Notice also what the gym fixes: the tasks, baselines, and metrics were all chosen by humans. The agent explores inside a box whose walls are the actual research judgment. That is not a criticism — it is what makes the benchmark scorable at all — but it locates the automation precisely: MLGym automates the inner loop and keeps the outer loop (what to work on, what to measure) in human hands. Hold that for §24.8.
Punchline: a benchmark for research is itself a research artifact, and the hard part is not the tasks but the verifier — Lecture 23's lesson wearing a new coat.
Minutes: 8–25 with §24.3, presenters leading. Steer: If presenters recite per-task results, redirect: "What did the designers decide that a SWE-bench designer never has to?" Force out the aggregation question and the fixed-walls observation. Ask the room: "Name one thing that becomes unmeasurable the moment a task has no known optimum." Distance to the ceiling — and with it, progress per unit budget, which is why §24.6's denominator matters. Expect confusion: Students file MLGym-Bench as "SWE-bench for ML." It is the opposite corner of the same axis: the same harness discipline, no oracle. Its 13 tasks are scorable, not solvable.
MLGym's headline finding, as the paper itself reports it: the frontier agents beat the provided baselines, but usually by finding better hyperparameters — learning-rate schedules, standard augmentations, known-recipe tweaks — and they fell short of producing novel hypotheses, algorithms, or architectures. Take both halves seriously. Beating research baselines across 13 tasks unattended is a real capability that did not exist three years ago. And it is a specific capability: search within a known method space, not extension of the space.
The paper frames agent capability as a ladder (its own taxonomy has finer rungs; here is the course's three-rung distillation):
Current agents on MLGym sit solidly at rung 2. The interesting question is what raises the ceiling, and the candidates sort themselves by how mysterious they are:
Note what the first lever implies: an agent that only tunes hyperparameters still benefits linearly from every cost optimization in Part II. You do not need to believe in rung 3 to care about the price of a hypothesis.
Punchline: "improves the baseline" and "does research" are different claims, and the gap between them is the most important sentence in the paper — the authors printed it themselves.
Minutes: shared with §24.2. Board: The three rungs, then "current agents: rung 2" — and next to it, "search throughput = budget ÷ cost/hypothesis," which is the bridge to §24.6. Ask the room: "Which rung is a leaderboard task like prefix-cache eviction on?" Rung 2 by construction — you are handed a trace, a baseline (LRU), and a metric. Then: "What would make an entry rung 3?" A policy that exploits structure nobody hand-coded, with the held-out evidence §24.5 demands. Expect confusion: Students read "beat the baselines" as refuting "no discovery." Say: "Both are true simultaneously; that is the finding. A grad student who only ever tuned hyperparameters would also beat many baselines."
The pipeline, end to end: idea generation → agentic tree search over experiments → figure generation with VLM feedback → manuscript write-up → LLM review. Two advances over v1 matter. First, v2 removes the human-written code templates v1 needed per domain — the agent writes its experiment code from scratch, which is what makes the claim "end to end" honest. Second, v1's linear iterate-and-revise becomes a tree: branch on ideas and variants, run the experiment at each node, score the node on its results, prune losing branches, expand promising ones — orchestrated by a dedicated experiment-manager agent that staggers the search through stages (roughly: preliminary investigation, then tuning, then the main research agenda, then ablations; read the paper's architecture section for its exact staging).
The tree is the systems-relevant design choice, because it converts research quality into a search-budget question. More nodes explored means more chances that some branch produces a publishable-looking result — the same "spend more inference, get better output" curve as the fan-out this course has already priced twice — Lecture 5's k ≈ 6 crossover on token cost, Lecture 10 §10.4's 5.7× on memory — except every sample now has a GPU experiment attached. That one structural fact generates both the economics of §24.6 (each node has a price, so the tree has a bill) and the statistics of §24.5 (the system's output is a maximum over the tree, so its headline results inherit selection bias by construction).
The VLM feedback loop is the second thing to notice: a vision-language model critiques the generated figures inside the loop — is the plot readable, does it support the stated claim — and the write-up is revised against that critique. The verifier has gone multimodal, but it is still a model.
The headline result: the authors submitted three fully autonomous manuscripts to an ICLR 2025 workshop, and one scored above the average human acceptance threshold — by their account the first entirely AI-generated paper to pass peer review. They then withdrew it before publication. Keep every clause of that sentence: three submitted, one accepted, workshop venue, withdrawn — each clause is a caveat, and §24.5 takes them in order.
Punchline: v2's real contribution is architectural — research as tree search with pruning — and its real result is a data point about peer review as a verifier, which is what we interrogate next.
Minutes: 25–42 with §24.5, presenters leading. Steer: If presenters dwell on the generated paper's content, redirect to the pipeline: "What flows through the loop, and who scores each thing that flows?" Get the tree on the board — nodes, pruning, and the three places a model sits in judgment (node scoring, figure critique, final review). Ask the room: "What does the tree buy over v1's straight line, for the same node budget?" Variance — and §24.5 explains why variance plus selection is exactly the dangerous combination. Expect confusion: "Pruning makes the search cheaper, so the tree is a cost optimization." It is a quality mechanism paid for in nodes: pruning reallocates budget toward promising branches, and §24.6 bills per node explored, not per branch surviving.
Three caveats, each a distinct verifier failure mode. Take them as a checklist for any future "an AI did science" claim, because there will be more of them.
1. The workshop bar. Workshop acceptance is the lightest form of human peer review: acceptance rates far above main-conference rates, fewer reviewers, less scrutiny, and an explicit mandate at many venues to accept preliminary or negative results. Passing the easiest human verifier is evidence — real evidence — but weak evidence, and the paper's own limitations section says as much. The claim "passed peer review" quietly ranges over a factor of several in stringency depending on the venue, and the demonstration chose the permissive end.
2. Cherry-picking across runs. The accepted paper is the best of a generated-and-selected set — three submitted, and the three themselves selected from the system's broader output. Reporting a max without reporting N is the same statistical sin as quoting best-of-N samples as model capability. Do the arithmetic once and it stays done: if a single attempt independently clears review with probability p = 0.05, then at least one acceptance occurs with probability 1 − 0.95^N — 14% at N = 3, 64% at N = 20, 99.97% at N = 158. An acceptance from unreported N is consistent with per-attempt quality anywhere down to p ≈ 1/N; the statistic is uninterpretable without the denominator. (Exercise 4 formalizes this, and §24.6 shows why N is also an economics number: a caching change alone moves the affordable N by 4.3×.)
3. Who verifies the verifier. Inside the loop, node pruning, figure critique, and pre-submission review are all performed by models. This is Lecture 23's central argument at full scale and it transfers without amendment, so spend the time on what is new rather than on re-deriving it. What is new is the failure's shape: not a paper full of gibberish, but a paper whose plots look right to a VLM, whose story reads right to an LLM reviewer, and whose experiment has a quiet leak — test data in the training set, a baseline run with the wrong flag — that no judge in the pipeline can execute the code to catch. And what is new is the artifact’s scale: in Lecture 23 the reward-hacked object was a patch that passed the tests; today it is a paper, and the audience it fools is people.
The one verifier that cannot be gamed cheaply is reality: rerun the experiment, hold out a test set the loop never saw, release the code and let strangers execute it. Lecture 23's measurement discipline is exactly that rule, and it carries over verbatim. Which is why MLGym-style benchmarks (executable metric, narrow scope) and AI-Scientist-style pipelines (model-judged narrative, open scope) are complements and neither suffices alone — one has a trustworthy judge for a toy question, the other a gameable judge for a real one.
So set the discussion question now and let it run under everything that follows: what result would convince you an agent did research? Candidates worth holding to the standard we just applied: an out-of-distribution prediction later confirmed; a method practitioners adopted without knowing its origin; a result on a task created after the model's training cutoff, verified by execution.
Punchline: peer review was designed to catch honest human error, not adversarial optimization pressure from a system that can draft a thousand papers for the price of a GPU-day — and §24.6 now prices exactly that sentence.
Minutes: shared with §24.4. Board: 1 − 0.95^N with the three values 14% / 64% / 99.97%. The one formula of the day outside §24.6, and it must still be up when §24.6 writes 158 and 675 beside it. Ask the room: "If you ran the venue, which of the three caveats would you fix first?" Usually "require the denominator" — cheap, mechanical, and the same ask Lecture 23 made of a benchmark score. Expect confusion: "The reviewers didn't know it was AI-generated, so the test was fair." Fair to the paper, yes — but blindness does not fix selection bias, and it raises the disclosure question in discussion seed 5.
The question "can agents do research?" has a systems dual: how many hypotheses per dollar? We now price an AI-Scientist-style run in the course's own units. Two modeling commitments, stated so they can be attacked. First, each tree node is itself a full agent trajectory — the agent edits code, launches the run, reads logs, analyzes — and we model it with Lecture 4's 20-step designer ledger: 97,000 prompt tokens submitted, 7,700 distinct (92% re-sent prefix; Lecture 13 §13.1 restates the ledger), and output ≈ 97,000 ÷ 60 ≈ 1,600 tokens, using Lecture 13 §13.5's 60:1 prefill:decode ratio. Second, we parametrize the tree at 20 nodes per paper attempt — our parametrization for round arithmetic, not the paper's configuration. Rates are Lecture 3's illustrative $3/$15 per Mtok (input/output).
Input: 97,000 × $3 / 1e6 = $0.291 — of which 92% (89,240 tokens, $0.268) is re-sent prefix Output: 1,600 × $15 / 1e6 = $0.024 Total ≈ $0.315 per node. Lecture 4's ledger, now denominated in hypotheses.
Tree search: 20 × $0.315 = $6.30 Write-up + VLM/review passes: ≈ 5 rounds × (30,000 in / 8,000 out) = 150,000 in ($0.45) + 40,000 out ($0.60) = $1.05 Total LLM bill ≈ $7.35 per attempt — the same order of magnitude as the roughly-$15-per-paper the AI Scientist authors reported for v1 (cite the order, not the figure).
Now the term students forget: each node also runs an experiment. At a modest 0.5 GPU-hour per node on an H100 at R = $3/GPU-hour:
20 nodes × 0.5 GPU-hr × $3 = $30 of experiment compute per attempt — 4× the entire $7.35 LLM bill.
The agent's thinking is cheap; its lab time is not. This inverts the intuition most people bring — that the frontier-model tokens must dominate — and it means the highest-leverage optimization for a research agent may be experiment scheduling: early stopping of doomed nodes, pilot runs before full runs. The LLM bill dominates only when the reasoning model is frontier-priced and the experiments are small — but it is the bill this course knows how to shrink, so shrink it:
Price cached input reads at 0.1× base — the standard cache-read discount, illustrative like the rates. Per node: 8% × 97,000 × $3/1e6 + 92% × 97,000 × $0.30/1e6 = $0.023 + $0.027 = $0.050 input (vs $0.291), so $0.074 per node (vs $0.315) — 4.3× more hypotheses per LLM dollar, from one optimization. The ceiling if cached reads were free: 97,000 ÷ 7,700 = 12.6× on input — Lecture 13 §13.5's prefix-reuse ceiling, now denominated in hypotheses.
A $1,000 LLM budget, counting the tree-search bill alone: uncached — 1,000 ÷ 0.315 ≈ 3,170 nodes ≈ 158 paper attempts; cached — 1,000 ÷ 0.074 ≈ 13,500 nodes ≈ 675 attempts. (The ≈$1 write-up tail per attempt trims these to ≈136 and ≈395 — the tail is not cached-discounted, so the ratio compresses to ≈2.9×, but the order-of-magnitude point survives.)
Feed 158 versus 675 into §24.5's arithmetic and the point lands: at p = 0.05 per attempt, 158 attempts yield an acceptance with probability 99.97% and 675 with near-certainty. A caching optimization changed what the acceptance statistic means. The verifier problem and the serving problem are the same problem seen from two ends — you cannot interpret a research agent's output without knowing its budget, and its budget is set by exactly the machinery Part II taught.
One more step closes the course. Lecture 1 §1.6 opened with the break-even T = 3D, and Lecture 2 pinned the constant: for the reference 7B at the Chinchilla ratio, cumulative inference compute overtakes the training run at T = 4.04e11 tokens — about 404 billion. Lecture 1 stated that in generated tokens and flagged the caveat that prompt tokens also cost ≈2N apiece, so count both here. An uncached attempt pushes 20 × 97,000 + 32,000 ≈ 1.97M forward-pass tokens through the model, so the crossover lands at 4.04e11 ÷ 1.97e6 ≈ 205,000 paper attempts — at $7.35 each, a $1.5M API budget. Not a hypothetical for a serious lab; a line item. 6ND was paid once; 2N per token is paid on every hypothesis, forever. A research org running agent loops is the mechanism by which inference stops being the product-serving bill and becomes the research bill.
Punchline: Part II of this course was, without ever saying so, a course in raising the number of hypotheses a dollar buys — and that number is now the denominator of every claim in today's papers.
Minutes: 20, instructor-led. Protect this budget; it is the last board work of the course. Board: Build the node price live — 97,000 × 3, then the ÷60 for output — box $0.315. Then the $30 experiment line directly underneath and circle the 4×; wait for the reaction, it is the board moment. Then the cached column: $0.074, 4.3×, and 158 → 675 attempts per $1,000. Leave 158/675 up next to §24.5's 1 − 0.95^N. Ask the room: "Your research agent is too expensive. Name three fixes, priced." Prefix caching (4.3×), capped tool outputs (exercise 1: 3.1×), cheaper experiment scheduling (the $30 term). Anyone who says "smaller model" gets asked what it does to p. Expect confusion: Students assume the LLM tokens dominate; the $30-vs-$7.35 split says otherwise, and exercise 3 makes the gap 680× for a self-hosted reasoner.
Reading-only; nothing here is new mechanism — it is Lectures 2, 3, and 5 composed once more, on the last workload of the course. Suppose you self-host the reasoning model (the reference 7B) instead of paying API rates. One paper attempt submits 20 × 97,000 = 1.94M prompt tokens, of which at most 20 × 7,700 = 154,000 are distinct — a deliberately pessimistic bound, since it credits no sharing at all between sibling nodes that in fact share the parent's whole transcript — plus 32,000 output tokens.
Prefill, at Lecture 2's illustrative 50% of dense BF16 peak: 2 × 6.74e9 × 1.94e6 = 2.62e16 FLOP ÷ 494.5e12 FLOP/s = 52.9 s. With perfect prefix reuse, only the 154,000 distinct tokens pay: 2 × 6.74e9 × 1.54e5 = 2.08e15 ÷ 494.5e12 = 4.2 s — the 12.6× ceiling again, now in seconds. Decode: 32,000 tokens × 4.0 ms = 128 s at batch 1. In Lecture 2's B = 116 GQA-8 configuration the card emits 116 tokens per 22.6 ms ambient step = 5,133 tok/s, so 32,000 ÷ 5,133 ≈ 6.2 s of stream share. About one minute of GPU reasoning per attempt, against 10 GPU-hours of experiments — roughly 600×.
Consistent with §24.6's dollar split, and sharper: for a cheap self-hosted reasoner, the tokens are nearly free and the experiments are everything. The token bill dominates only at frontier API pricing — which is why §24.6 used it.
The workload shape is Lecture 5's agent-serving profile taken to its limit. By tokens it is ~98% prefill (1.94M in, 32k out). It is ~92% re-sent prefix — and better than that: a tree expansion fans out sibling nodes that share a long common prefix (the parent's entire transcript), which is ideal RadixAttention-style prefix-cache traffic (Lecture 13 §13.3), more regular than anything in Parrot's 3/94/72/99% app profiles (Lecture 13 §13.9). And it idles in exactly Lecture 5's worst way: while a node's experiment runs for minutes or hours, its session is stalled — the 13.4 GB stalled-session problem at experiment timescales, which the Nov 23 meeting turned into a three-way choice between keeping, swapping, and discarding. Here it is not a close call. Lecture 19's keep-versus-discard break-even is 15.1 s at a 4,096-token context and 21.3 s at 16,384; an experiment stall is one to three orders of magnitude past that, so discard the KV and keep the text. Rebuilding an average 4,850-token node context costs Lecture 13 §13.5's 157 ms of prefill against an hour of experiment — noise.
Every Part II lever, one line each, on this one workload: prefix caching (Nov 4) — up to 12.6× on input tokens, the single biggest lever here; routing (Oct 26) — send sibling nodes to the replica already holding their shared parent prefix, which is Lecture 13 §13.6's cache-locality argument at tree granularity, and a node's ≈4,850-token average prefix is long enough to be worth chasing; quantization (Nov 9) — halve b and roughly double the concurrent node evaluations per card; disaggregation (Oct 14) — a workload that is 98% prefill by tokens wants a prefill-heavy pool, and a decode-optimized pool is wasted on it.
Punchline: an "MLGym for LLM serving" would look like this course — baseline = Lecture 2's 29-sequence / 0.054·R configurations, metric = $/Mtok or hypotheses per dollar, and a prefix-cache eviction task is exactly one of its tasks — which is also the readiest final project on offer. That is discussion seed 1, handed to you.
Minutes: 0 — reading-only, not scheduled. If §24.4–24.5 finish early, do the worked block only and write "1 minute of thinking, 10 hours of lab time" under it. Ask the room (if used): "Which Part II lever would you pull first here?" Prefix caching — then make them say why routing is the second pull, not an independent one: locality is what makes the cache hit.
Three residuals, argued rather than asserted — and honestly labeled as claims about current systems, not theorems.
Problem selection. MLGym's 13 tasks, their baselines, and their metrics were chosen by humans; AI Scientist-v2's idea seeds and topic areas were chosen by humans. Neither loop contains the step "decide what is worth asking" — not because it is mystical, but because no current system even represents it, let alone optimizes it. The automation starts one step after the most consequential decision.
Taste. Pruning on a metric works when the metric is the goal. Most research value is recognizing which anomaly matters — this course's own material is the example: Lecture 2 §2.15's "29 versus 295" is just two numbers until someone recognizes that the gap between what the memory permits and what the hardware demands is the field of LLM serving. No verifier in either paper scores "this discrepancy is a thesis." A model can be prompted to hunt anomalies; ranking them by consequence is the part that still looks like taste.
Verifier design. Lecture 23's lesson, now the course's closing claim: the durable human job in an agentic world is specifying what counts as success and building the harness that measures it honestly — against an optimizer that will exploit every gap you leave. Assignment 5 is this job in miniature, and a held-out trace in a project's evaluation is the same idea compressed into one rule. It is also the job that gets harder as agents get stronger, because the optimization pressure against your harness rises with the strength of the optimizer.
Direction of travel, stated plainly: the ladder has visibly moved one rung in about two years, and each residual above is where today's systems stop, not where systems must stop. The scientific move is to write down now what evidence would change your mind — a confirmed out-of-distribution prediction, an adopted method of machine origin, a rung-3 result on a post-cutoff task — so that you recognize it if it arrives, instead of re-fitting your definition of research to exclude whatever machines just did.
Punchline: this course taught you to price the machine's work; what it cannot price is deciding which work is worth doing — and that was the part you were doing all along.
Minutes: 62–70 with the seeds; keep this to the three headings and one sentence each if time is short. Ask the room: "Which of the three residuals falls first?" Push for mechanisms: verifier design is under direct optimization pressure (adversarial), taste may be learnable from preference data, problem selection has no training signal anyone has collected. There is no consensus answer; that is the point. Expect confusion: "What stays human" gets heard as reassurance. Reframe it — each residual is a job description, and the third one is graded on Dec 2.
The calendar, one line each, in date order — these dates are firm:
Grading recap: assignments 10% each for 1, 2, and 4, 18% each for the optimization pair, 3 and 5, and 4% for 2.5, final project 12%, paper presentation 12%, participation 6%, plus up to 10% bonus. There is no exam; the project is what closes the semester.
R base; anti-gaming rule = a held-out trace the agent never sees. §24.7 argues this course is most of the gym already — what is missing?N output. What does your answer do to peer review's value as a verifier?Minutes: 70–75 for the logistics block — announcement, not content, but it is the last topic class; never cut it. Seeds 1–3 run in the 62–70 synthesis slot; seeds 4–5 are spillover. Board: The four dates in a column. Leave them up as they walk out — the last thing on the last board.
N with N mostly unreported: at p = 0.05, best-of-20 clears review 64% of the time. Workshop bar, cherry-picking, model judges: three distinct verifier failures.| Quantity | Value | Source |
|---|---|---|
| LLM cost per experiment node (20-step trajectory) | $0.315 uncached / $0.074 cached | 97,000 in + 1,600 out at $3/$15; 0.1× cached reads |
| LLM bill per 20-node paper attempt | ≈ $7.35 ($6.30 tree + $1.05 write-up) | §24.6 |
| Experiment compute per attempt | ≈ $30 — 4× the LLM bill | 20 × 0.5 GPU-hr × $3 |
| Prefix-caching multiplier / ceiling | 4.3× / 12.6× hypotheses per LLM dollar | $0.315→$0.074; 97,000/7,700 |
| Attempts per $1,000 (tree bill) | 158 uncached → 675 cached | 1,000 ÷ (20 × node cost) |
Best-of-N acceptance at p = 0.05 | 14% (N=3), 64% (N=20), 99.97% (N=158) | 1 − 0.95^N |
| GPU reasoning vs experiments per attempt, self-hosted 7B | ≈ 1 minute vs 10 GPU-hours (≈600×) | §24.7 |
| Research workload shape, per attempt | ≈98% prefill by tokens (1.94M in / 32k out) | §24.7 |
| Stalled node session | discard — break-even 15.1–21.3 s, experiments stall for minutes | Lecture 19; §24.7 |
T = 3D crossover, reference 7B, in attempts | 4.04e11 tokens ≈ 205,000 attempts | 4.04e11 ÷ 1.97e6 |
T = 3D return here?6ND is paid once; 2N per token is paid on every hypothesis forever. For the reference 7B the crossover is 4.04e11 ≈ 404 billion forward-pass tokens, i.e. ≈205,000 attempts at ≈1.97M tokens each — a ≈$1.5M API budget, well inside a serious lab's spend. A research org running agent loops is how inference becomes the dominant compute bill: not by serving users, but by doing science. Lecture 1's forecast, closed as a line item.p = 0.05. Compute P(≥1 acceptance) for N = 3, 20, 158, 675; find the N at which one acceptance is the expected outcome; and say why N must accompany any acceptance claim. Solution sketch: 1 − 0.95^N = 14% / 64% / 99.97% / ≈100%. Expected acceptances = Np = 1 at N = 20. An acceptance with unreported N is consistent with per-attempt quality as low as p ≈ 1/N — the number is uninterpretable without its denominator, and §24.6 showed a caching change alone moves the affordable N by 4.3×. Any claim of the form "an agent's paper was accepted" is a claim about a ratio whose bottom half is missing.R to ≈0.049·R; requiring the conversion prevents optimizing a proxy the bill ignores. Gaming risk — memorizing the public trace; rule — score on a held-out trace. The line: tuning LRU-K's K or TTLs = rung 2 (improve); a policy that exploits structure nobody hand-coded — say, tree-sibling prefix reuse in agent traffic (§24.7) — is approaching rung 3, and you should demand the held-out evidence §24.5 demands.Required — MLGym. Read the introduction for the capability-ladder framing and the benchmark-design section for how the 13 tasks, baselines, and compute budgets are specified — that machinery is the durable contribution. Read the results and analysis for the honest headline: improvements come usually from better hyperparameters, not novel hypotheses or algorithms — and check how scores are aggregated across attempts (best attempt versus average; it decides what the leaderboard means, and it is the same max-versus-distribution issue as §24.5's arithmetic). Skip the per-task appendix details. Hold this question: what, concretely, would an agent have to output on one of these 13 tasks for you to call it rung 3?
Required — AI Scientist-v2. Read the architecture section for the agentic tree search — branching, node scoring, pruning, the experiment-manager's stages — and the VLM feedback mechanism; read for what flows through the loop and who judges each thing that flows. Read the workshop-submission account and the limitations/ethics discussion carefully; the caveats section is the most load-bearing part of the paper. Skim one generated manuscript in the appendix to calibrate the workshop bar with your own eyes. Skip the remaining appendices. Hold this question: if you were the optimizer, which verifier in this pipeline would you attack first?
Optional — Automated statistical model discovery. Read for Box's loop as the classical template and for how a formal language — probabilistic programs — makes every proposal mechanically checkable. The contrast with free-form paper generation is the point; skip the domain case studies. Hold: what does the formal language buy that an LLM reviewer cannot?
Optional — SWE-bench. Re-read only the task-construction section, as the cleanest closed verifier on record: issue + repo + hidden tests. Hold: what breaks, step by step, if you try SWE-bench's construction for research tasks?
This note is the end of the chain, and there is no topic lecture after it: the semester's remaining meetings are student sharing IV on Monday, November 30 and Wednesday, December 2, with no class on Wednesday, Nov 25. So the map points at deliverables instead, in date order.
N maxima. Write the one the AI Scientist cannot yet write.Lecture 1 opened with T = 3D as a forecast about where the compute would go. Lecture 24 leaves it as a bill — one you now know how to read line by line, and, more to the point, how to shrink.