Assignment 3 — Optimize the agent (18%)

Out Mon Oct 5 · due Sun Oct 25, 11:59pm · individual · all five assignments

Your Assignment 2 agent works, and you know what it costs. Now make it cheaper and faster without making it worse. You are still renting someone else's serving stack, which is the whole point of doing this now: every lever you have is a harness lever, so whatever you win here is unambiguously yours rather than the provider's.

That matters for what comes next. Assignment 4 takes the frontier model away and Assignment 5 hands you the serving system underneath it. If you have already wrung out the harness, you will know which of those later wins came from owning the stack and which you could have had all along.

Five metrics, fixed for the semester

Assignments 2, 3, 4 and 5 repeatedly measure the same system, so the measurement contract must not change. You will track task success rate, latency, cost per task, tokens per task, and steps per task. The exact definitions are established in Assignment 2. Once fixed, they stay fixed.

This assignment optimizes cost and latency while holding success rate. Assignment 4 measures the same five metrics after replacing the frontier model and also adds first-try schema validity. Assignment 5 measures the system again with the serving stack exposed and adds a serving-side profile. If the workload or success criterion drifts between assignments, the comparison is no longer meaningful.

1. Profile before you touch anything. The five metrics tell you what your agent costs; they do not tell you where it goes. Break the cost down until you can name the expensive part: input versus output tokens per step, how much of each request is context you already sent, wall-clock split across model time, tool time and retries, and the step histogram rather than just its mean. A cost you cannot attribute is a cost you cannot remove.
2. Rank the levers, in writing, before you pull them. Which one you expect to be the biggest win, why, and roughly how much. You will be wrong about at least one, and noticing that is worth more than being right.
3. Optimize. Every lever here lives in your code:
   • Send fewer tokens. Stop re-sending context that has not changed, summarize or window the history, trim tool output before it enters the transcript, drop the parts of your system prompt that are not earning their place.
   • Pay less for the tokens you do send. Providers charge a fraction for cached prefixes — but only if your prefix is actually stable, which is a design decision about how you order your prompt, not a flag you set.
   • Make fewer calls. Earlier stopping, fewer retries by making malformed output less likely in the first place, collapsing two steps the model can do in one.
   • Overlap what you can. Independent tool calls do not need to be serial. This moves latency without moving cost, which is why you report them separately.
   • Match the model to the step. Routing the easy steps to a cheaper model is a real optimization; it is also the one most likely to cost you quality, so it needs the held-out check below.
   • Design the tools better. A tool that returns 400 lines when the model needed one number is a token problem disguised as an interface.
4. Re-measure after each change, not just at the end. Same task set, same five metrics, same scorer. One aggregate before-and-after cannot tell you which change paid, and some of them will not.

What you may not do: change the task set, loosen the success criterion, or drop the tasks that were failing anyway. Those three are how an optimization result gets faked, usually without meaning to. Switching to a cheaper model is fair game and interesting; switching to a better one is not an optimization.

Then say what you could not see. You have just optimized a system whose expensive half is invisible to you: you cannot see prefill separate from decode, you cannot see how your request was batched with strangers', and you cannot see whether your cached prefix actually hit. Write down the questions you could not answer from outside the API. Part II is the answer to that list, and Assignment 5 is where you get the instruments.

This one is written as a blog post

Assignment 3's write-up is a published blog post, not a PDF — you open a pull request against the course blog repository and it goes live under your name once it is reviewed and merged. The mechanics and the deadline rule are below; read them before you start writing.

Write it for somebody who did not take this course. That constraint is doing real work here: the reason this assignment gets a post rather than a report is that its result is an argument — you picked the levers, you ranked them in writing before pulling them, and the interesting part is where the ranking was wrong. A reader who has to be told what a KV cache is will not follow a bullet list of deltas, so you have to explain what you were trying to do and why the numbers mean what you say they mean. That is the same discipline as stating a denominator, applied to prose.

The post should carry the profile, the ranked prediction as you wrote it down beforehand, what each change actually bought on held-out tasks, and the levers that did nothing — that last one is where most of the value is, and the section a report format quietly lets you omit. You may reuse your Assignment 2 report for the setup: the reader needs to know what the agent is before they can care that it got cheaper. Reusing your own figures, measurements, or explanatory text is expected; state what you reused, because the grade still comes from the new work and the new analysis.

How the 18% breaks down: the profile, detailed enough to name where the cost is (4%); the ranked prediction, judged on the reasoning rather than on whether it turned out right (3%); measured improvement with success rate held on held-out tasks, reported per change (8%); and the blog post, including the levers that did nothing and the questions you could not answer from outside (3%).

Publishing the optimization work

The post is actually published. The course will maintain a repository, and you will contribute your post through the same workflow used by an open-source project: fork the repository, add your post, open a pull request, respond to review, and merge.

The repository URL will be posted here, on the course home page, and on Canvas before Assignment 3 opens.

1. Write the post. Add one Markdown file under the repository's posts directory using the naming convention in the README. Put figures beside the post and keep them reasonably sized. Do not commit raw traces or giant image files.
2. Open the pull request by the assignment deadline. The deadline is based on when the PR is opened, not when it is merged. Review happens afterward. A PR opened before the deadline remains on time even if the review process takes several days.
3. Respond to review. Expect editorial and factual comments: unsupported claims, missing denominators, unclear figures, unnecessary paragraphs, or results that need qualification. Being asked to revise is normal. Ignoring review is not.
4. Merge and publish. Once accepted, the post goes live with a public URL. It must also include the same AI-use note required for other submitted work: what you delegated, what you kept, one place the agent helped, and one place it misled you. By the end of the semester, you should have a public technical record of building and optimizing an agent stack, not only PDFs sitting in Canvas.