Assignment 4 — Serve your own agent (10%)

Out Mon Oct 26 · due Tues Nov 10, 11:59pm · individual · all five assignments

Your Assignment 3 agent works, and by now it is lean. Some of that is your harness and some of it is a frontier model quietly covering for your harness — and from where you are standing you cannot tell which. This assignment takes the frontier model away.

Serve an open-weight model yourself, on an engine you run — vLLM, SGLang, whichever you like — and point your optimized agent at it. Plan on a 7–8B class model, whose bf16 weights and a modest serving cache fit on a single 24GB GPU. Larger is allowed if you have the hardware, but it is not the point: the smaller the model, the more the harness has to do, and the harness is what is being graded.

The GPU comes from the course's gpu-cs2680 partition — one srun and you have an interactive shell on a GPU node. Your research group's card is fine too; report which one you used either way, because none of these numbers are comparable across hardware.

1. Stand it up. Get the model served behind an OpenAI-compatible endpoint and your agent talking to it. Report everything that broke on contact — the chat template, the tool-call format, JSON that will not parse, stop tokens that never fire, a context window your prompt no longer fits.
2. Measure the drop. Same task set as Assignments 1 and 3, the same five metrics, same method — with cost now read off your own GPU-seconds rather than a price sheet. Report the success-rate delta. It will be large. That number is not a failure, it is the assignment.
3. Add the one metric that only exists now. The fraction of model outputs that parse and validate against your tool schema on the first try. Report it, and report the failure breakdown beside it. Constrained decoding, tighter schemas and retry-with-repair all move this number, and it is what separates “the model cannot do the task” from “the model cannot emit the format” — a distinction the frontier model was quietly hiding from you.
4. Recover it in the harness. Constrained or structured decoding, tighter tool schemas, decomposing a step the model cannot do in one shot, retry-with-repair on malformed output, an explicit verification pass. What you may not do: reach for a bigger model, or fall back to the commercial API. The question is how much of the frontier model's advantage was doing work your harness should have been doing.
5. Report the recovery curve. Success rate, first-try validity rate, and the failure breakdown after each change, not just at the end. One aggregate number cannot tell you which intervention paid, and several of them will not. Carry cost and latency along in the same table: an intervention that buys five points of success rate by tripling the step count is a result, not a free win.

Cost and latency are worth recording here, but they are context rather than the deliverable: this assignment is graded on getting quality back, and Assignment 5 is where you make the stack underneath it cheap.

The write-up is a PDF report submitted through Canvas, compact and technical: what you built, how it works, how you evaluated it, what failed, and what the measurements show.

The quality you recover here is the threshold Assignment 5 holds still, so this report is the setup for your Assignment 5 blog post — write it once, properly, and reuse it there.

How the 10% breaks down: the bring-up, with evidence it actually runs (2%); the baseline comparison against Assignment 3 on the same tasks and metrics (2%); harness interventions with per-change measurement (4%); and the write-up, including the interventions that did not help (2%).