Assignment 2 — Design an agent (14%)

Out Mon Sep 21 · due Sun Oct 4, 11:59pm (agent loop and problem set) · individual · all five assignments

Replace claude -p with an agent loop you write yourself against a commercial API — OpenAI, Anthropic, or Gemini, your choice. Do not use an agent framework: you own tool calling, context management, retries, error recovery, and the stopping condition. Keep the console from Assignment 1 as the front end and promote its example tasks into a fixed evaluation set. A small agent you can explain and measure is more valuable here than a capable one assembled from parts.

Two different tools, and it is worth being clear about which is which. You build with Claude Code, as in every assignment. The API your agent calls is a separate decision and stays entirely yours — requiring Claude Code says nothing about which provider your own loop talks to. If you do point your agent at Anthropic's API, keep its credential out of ANTHROPIC_API_KEY: once that variable is set, Claude Code prefers it over your Harvard login, so an exported key silently bills your own account and muddles the session files you submit. Name your agent's variable something else, or load it from a .env your agent reads rather than exporting it globally — and unset ANTHROPIC_API_KEY puts you back on the subscription.

The write-up is a PDF report submitted through Canvas, compact and technical: what you built, how it works, how you evaluated it, what failed, and what the measurements show. Write it as though somebody will read it in October, because you will: your Assignment 3 blog post has to introduce this agent to a reader who has never seen it, and reusing this report for that is expected rather than frowned on.

This assignment carries a second, smaller task on top of the loop: write four problems that an agent can be asked to solve, one easy, two medium, one hard, each with a reference solution and an automatic scorer — the contribute task below. The class's submissions are pooled and a selection becomes the held-out grading set that Assignment 3, Assignment 4 and Assignment 5 are measured against, so the two parts of this assignment are the harness of the whole back half of the course. Leave your agent in a state you can re-run, because calibrating your problems is done with this agent.

Instrument it from the first commit: per-step token counts, latency, tool time, retries. Assignment 3 optimizes this exact system and Assignments 4 and 5 take it apart, and you will be glad the measurement is already in place rather than bolted on afterwards.

Report five numbers. These are the assignment's headline result, and Assignments 3, 4 and 5 are measured against them — so define them now, precisely, and do not move them afterwards:

Metric	Report, exactly	Why this definition
Task success rate	The fraction of tasks completed correctly, scored by a script in your repo against a criterion you wrote down before the first run. Give the count as well as the percentage.	A criterion written after you have seen the output is not a measurement. Fix the task set first — twenty or more, varied enough that one lucky prompt cannot carry the score — and version the scorer, so the same tasks can be re-scored the same way in October.
Latency	Wall-clock per task, from the request arriving to the final answer: mean and P90.	Agent latency is heavy-tailed, and one task stalling in a retry loop hides inside the mean. The mean is what a batch of tasks costs you in time; the P90 is what a user actually feels. The gap between the two is itself the finding.
Cost per task	Dollars per task at the provider's posted prices, with the price sheet you used stated once. In Assignments 4 and 5 this row means your serving cost — GPU-seconds per task times a $/GPU-hour rate you fix now and keep.	A dollar figure alone expires the next time a provider reprices, and it is not defined at all once you serve the model yourself. Pin the rate you divide by and the number stays comparable across every assignment that reports it.
Tokens per task	Input and output tokens per task, counted separately, mean and total — including tokens spent on retries and on outputs you threw away.	Prices change; token counts do not, so this is what lets anyone re-price your agent later, including you in Assignments 3 and 5. Input and output differ in price by roughly an order of magnitude and are moved by different fixes — re-sent context versus verbosity — so a single total hides the one you can actually act on.
Steps per task	Model calls and tool calls per task, counted separately, mean and P90. A retry is a step.	This is the harness's own number, and the one Assignments 3 and 5 attack directly. Cost and latency are largely a consequence of it, so a step count you cannot explain is a system you cannot optimize.

And one table that is not a number. Alongside the five, break your failures down by cause — wrong answer, malformed tool call, ran out of steps, provider error, harness crash. A success rate tells you how often you failed; only the breakdown tells you what to fix, and Assignment 4 is essentially a whole assignment spent choosing which of these buckets to attack.

Then say where the tokens and the wall-clock actually go — which step, which tool, which retry. You are renting someone else's serving stack right now and cannot see inside it; note which questions you cannot answer from the outside. Those are the questions Part II is about.

How the 14% breaks down: the agent loop, written without a framework (3%); the five metrics above and the failure breakdown, measured over a fixed task set with the success criterion written down in advance (4%); the write-up, including the token and wall-clock accounting and what you could not see from outside the API (3%); and the contributed problem set (4%).

These five numbers outlive this assignment. Assignment 3 holds success rate fixed while driving cost and latency down, Assignment 4 re-measures all five after taking the frontier model away, and Assignment 5 does the same once you own the serving stack too. A task set or a success criterion that shifts in between makes every one of those comparisons meaningless — and the drift is rarely deliberate, it is a task quietly reworded or a scorer loosened to stop a failure being annoying. Freeze all three here and version the scoring script with the code.

Submit three artifacts: your code, your write-up as a PDF report through Canvas, and an archive of your Claude Code session files from ~/.claude/projects/. Session history expires after 30 days by default, so archive before the deadline rather than after — and because those files record every file the agent read and every command it ran, scan them for credentials first.

Task: contribute the problem set

Write four problems — one easy, two medium, one hard — that an agent can be asked to solve, each with a reference solution and a way to check an answer automatically. Run all four through the agent you just built. At least two must pass. A problem here is exactly the shape of thing your evaluation set already holds: a task statement handed to an agent, and a criterion that says whether the agent's output was right. Four requirements, and a submission that misses one is not usable:

Self-contained. It must run against anyone's agent, not just yours. No dependency on your repo layout, your private tools, your API keys, your machine, or a service that needs an account. If the problem needs input data, ship the data — small enough to commit, and in the submission.
Machine-checkable. Ship a scorer: a script that takes the agent's output and exits 0 or 1. Exact match, a parsed number within a stated tolerance, a test suite the produced code must pass, a regex over the final answer — whatever fits, as long as no human judgment is in the loop. State the output format the agent is expected to produce, because a correct answer in an unparseable shape is a scoring bug you are handing to us.
Definitely answerable. There is one right answer, or a clearly bounded set of them, and you know it. A problem is not hard because it is ambiguous, underspecified, or a matter of taste — those are broken problems, and they are the most common failure mode of this assignment.
In this course's subject area. Agents, serving systems, GPUs, kernels, cost and performance reasoning — the material of the course. A hard Sudoku is not on topic.

Difficulty is defined against your own agent, measured, not guessed. Run each problem five times and report the pass count, against the same fixed task-set discipline as your success rate above — one draw is not a measurement.

Tier	Your agent, over five runs	What it is testing
Easy ×1	5 of 5. It should not be able to fail this one.	That the harness works at all: tool call, one or two steps, correct format. This is the tier that catches a broken agent rather than a weak one, which is exactly what you want in a grading set.
Medium ×2	2–4 of 5. Passes, but not reliably.	Several dependent steps, or a tool result that has to be interpreted rather than copied. The two mediums should fail for different reasons — say which, in one sentence each. Two problems that break the same way are one problem.
Hard ×1	0–1 of 5. Your agent essentially cannot do it.	Where the ceiling actually is: long horizons, a trap the model walks into confidently, arithmetic it fakes, or state it has to keep across many steps. Still definitely answerable — you must supply the correct answer and be sure of it.

The submissions are pooled across the class, and a selection from that pool becomes part of the grading task set for Assignments 3, 4 and 5. That is the point of the exercise, and it is why the bar is machine-checkability rather than cleverness: a problem nobody can score without reading the transcript by hand is a problem we cannot use. It is the one part of the course where you are on the other side of the instrument — every other assignment asks you to measure an agent; this task asks you to build the ruler.

Your submissions become course material, and that cuts in your favour. We pool every student's four problems, select from the pool, and use the selection to grade Assignments 3, 4 and 5 — so the set you are measured against later is drawn mostly from problems other people wrote. Tuning your agent to your own four buys you almost nothing. Writing four problems that probe something real, and reading the tiers as a description of where agents actually break, is the move that pays. Assume your problems may be used, may be shown to the class after grading, and may be rewritten for clarity; if you have a reason something should not be reused, say so in the write-up and it will not be.

← Assignment 1 — Use an agent Assignment 3 — Optimize the agent →