Out Wed Nov 11 · due Tue Nov 24, 11:59pm · individual · slides · all five assignments
Assignment 5 builds on Assignment 4. Its goal is a strong agent of your own. Start from your Assignment 4 agent, fix what held it back, and take good ideas from other agent designs. You may also change the model and the serving engine, but you do not have to. You still serve on one RTX PRO 4500 (32 GB).
The same agent runs Terminal-Bench and Assignment 4's tasks, so first generalize your Assignment 4 agent.
⚠️ ONE RTX PRO 4500 AT A TIME.
As in Assignment 4, use at most one GPU at a time. The first violation gets a warning, and each further one costs 10% of your score on this assignment.
⚠️ THE LEADERBOARD DOES NOT ACCEPT LATE DAYS AND STOPS TAKING SUBMISSIONS ON TUESDAY, NOVEMBER 24, AT 11:59 P.M. ET.
Without a submission by then, you get 0 of the 50 leaderboard points and no bonus. Late days apply only to the blog post and the video.
⚠️ WRITE YOUR BLOG POST YOURSELF.
If we find that your blog post was written by AI, we will deduct points.
Download the Assignment 5 starter archive into the root of your course repository, extract it, and run bash assignment5/starter_code/init.sh before you start work with Claude Code, as in Assignment 4. Copy your Assignment 4 src/ and engine/ into assignment5/starter_code/. The starter's run_all.py runs your agent on Terminal-Bench 2.0 or on the 25 tasks of Assignment 4's starter.
On the cluster, Assignment 5 has its own course environment, which is Assignment 4's environment with two changes: the model is served as cs2680-model, and engine.json may name a model folder of your own. Set it in every new shell, and use $A5 wherever Assignment 4's page uses $A4:
export A5=/shared/courseSharedFolders/176914outer/176914/a5 # the course environment
The A4_* variables that launch.sh gets, such as $A4_PORT, keep their names, and so does the folder ~/a4. When you connect your agent as in Assignment 4, set all three CS2680_MODEL_* variables to cs2680-model, as the starter's README.md shows.
Terminal-Bench 2.0 is a public benchmark of 89 tasks, from software engineering to system administration, security, data science and model training. Each task gives your agent an instruction and a container with internet access, and sets its own time limit, CPUs and memory. When your agent finishes a task, the task's hidden tests run in the same container and check its final state. The images take about 49 GB compressed and run only on x86_64. Public results for version 2.0 are on its leaderboard on Harbor Hub.
The dispatcher gives Terminal-Bench tasks and Assignment 4's tasks in the same form, so your agent needs no code for a particular benchmark. Three things change from Assignment 4:
next_task() returns task (the index k), instruction (the whole task as text), sandbox_url, workdir and time_limit_s. time_limit_s is the most seconds the task may stay open: the smaller of its own limit and what is left of the run's limit (for Assignment 4's tasks, what is left of the 4 hours). It is None only when no limit applies, as for an Assignment 4 task in a local run with A3_RUN_LIMIT_S=0.done(k) ends task k, and the course grades the task's final state. A task still open when time_limit_s runs out is graded at that moment.evaluate, extract_patch, submit_patch and continue_task are removed.To generalize your Assignment 4 agent, make seven changes:
instruction. Remove prompts and code that assume a repository to fix or a patch to produce. The working directory may not be a git repository, and tools and prompts written for code repositories help less on most Terminal-Bench tasks.evaluate(k), which in Assignment 4 ran a task's hidden tests on your agent's patch before the task ended, your agent must write and run its own checks before done(k).done(k), so start it detached, with its output redirected to a file.src/sandbox_server.py refuses paths outside the repository and /work, but tasks may write anywhere, such as /etc. Use the starter's copy, or remove the limit from yours.time_limit_s.CS2680_MODEL_* variables, which the leaderboard sets to cs2680-model. Do not hard-code Assignment 4's model name. Your server no longer lists it, and some engines, such as vLLM, reject a request for it.Work as in Parts 2 and 3 of Assignment 4: measure, find what limits your run, and fix it. In the agent, study the tasks it fails and try ideas from other agent designs. In the serving engine, continue your Assignment 4 work within its allowed changes. A faster server also gives your agent more turns within each task's time limit. Results vary between runs, so run a change more than once before you conclude that it helped. Record each mistake your agent makes and each technique you try, including the ones that did not help.
You may also change the model: serve another checkpoint, quantize, prune or distill the model, train or fine-tune one, or add a draft model for speculative decoding. Assignment 4's rules that fix the model do not apply. Your server still serves the name cs2680-model and must be ready within 45 minutes, including loading the weights. You may train a model in any way, except on the tasks and your agent's runs of them (see the rules below), and on the course cluster within the one-GPU limit. A smaller model can solve fewer tasks, so measure tasks solved after every model change.
To serve another model, put it in a folder of your own on the cluster, under /shared/courseSharedFolders/176914outer/176914/a4-models/<your cluster username>/, and set engine.json's "model" to the folder's path. The folder holds everything your server loads, a draft model included, in at most 48 GiB. Each grading copies it to the GPU node when the grading starts, and the copy counts toward the 45 minutes. The leaderboard's Engine conventions page lists the folder's rules, and $A5/bin/check-submission.sh --launch tests a folder as a grading uses it.
Your harness should work across general tasks rather than being tailored to specific tasks or repositories in the benchmarks. The following practices are considered violations. Any submission containing them will be invalidated and excluded from the leaderboard, and your ranking will revert to those of your best valid submission:
__init__" is not allowed.The harness may instead guide the agent to inspect relevant source code, examine nearby or similar structures, and derive an appropriate solution during the evaluation. Any task-specific insight should be discovered by the agent as it works and can be verified through the trajectory.
You may run Terminal-Bench 2.0 and the 25 tasks of Assignment 4's starter yourself, but not the 26 leaderboard tasks, to measure changes, tune general settings such as server flags, and fix general mistakes like those in Part 1.
We keep and review the trajectories, source code, and patches generated during each submission evaluation. If we find a violation, we'll let you know by email. If you're unsure whether your harness design follows these guidelines, feel free to email us before submitting.
You submit three things:
assignment5/starter_code/: your agent in src/ and your server in engine/, zipped into submission.zip for the leaderboard, and your agent sessions packaged with python3 package_agent_sessions.py.The leaderboard is at https://leaderboard.cs2680.com/a5, and you log in with the same account as in Assignment 4. As in Assignment 4, it starts your server on a fresh RTX PRO 4500 node and runs your agent. Every grading of your submission.zip has two runs: Terminal-Bench 2.0 first, then the 26 tasks, which start only after the Terminal-Bench run has ended. Your server starts once and serves both runs. Each run is measured on tasks solved and end-to-end time:
Each run's limit and end-to-end time count from the run's start. In both runs, the tasks come in a new random order in every grading, and a task never opened counts as not solved. The course uses the two results to study how much the agent harness changes a benchmark score and how to control the variance between runs. You get 2 gradings a day, and Assignment 4's other leaderboard rules apply, with two additions for the two runs:
Write a post on the course blog about what you learned from Assignments 2-5. You do not need to organize it by assignment.
Before you start, think back over all four assignments and write about as many of your lessons as you can think of. For each lesson, describe what you did, what happened and what you learned, with your own measurements and examples. Mistakes are lessons too, so include them along with what worked.
Write the post yourself, in your own words. Don't generate it with AI. Publish it with the blog's publishing guide. The post is public, so make sure there is no private data.
We expect everyone who puts in enough effort to get full or nearly full marks.
Baseline. A submission meets the baseline if its Terminal-Bench 2.0 run reaches either of these two targets:
| Target | Tasks solved | End-to-end time |
|---|---|---|
| Fast | at least 20 | at most 2 hours |
| Accurate | at least 30 | at most 6 hours (the whole run) |
Both numbers describe one whole run: the tasks it solved and its end-to-end time. As in Assignment 4, the end-to-end time runs from the run's start, when your server is ready, to the moment the last task ends, whether by done(k), at a time limit, or when your agent stops. For example, a run that solves 22 tasks and whose last task ends 2 hours and 10 minutes after the start misses the fast target, even if it had solved 20 tasks within the first 2 hours.
Leaderboard score. If s is the most tasks one of your valid submissions solved on Terminal-Bench 2.0:
score = 50 if any of your valid submissions meets the baseline
score = min(2 × s, 40) otherwise
For example, if none of your submissions meets the baseline and your best one solved 15 tasks, you get 30 points. A submission that solved 20 or more tasks but missed both targets gets 40. Your score depends only on your own submissions, not on other students' results.
Terminal-Bench ranking. The leaderboard ranks every valid submission from all students as in Step 4 of Assignment 4's leaderboard, with 20 tasks in place of 7: submissions that solved at least 20 tasks are ranked in rounds over tasks solved and end-to-end time, and students with no valid submission that solved 20 tasks rank below all of them, by the most tasks one of their valid submissions solved. Your rank does not change your score, and the bonus uses its own rankings.
The bonus adds up to 5 percentage points to your final course score: up to 3 from the run on the 26 tasks and up to 2 from the Terminal-Bench run. Only submissions that meet the baseline are qualified for bonus, so the techniques behind a bonus must also work on Terminal-Bench. The baseline and the bonus must come from the same submission: a submission that does not meet the baseline itself earns no bonus, even if another of your submissions meets it. Each of the two runs has its own ranking of qualified submissions, in rounds as in the Terminal-Bench ranking above, and the two rankings may use different submissions of yours. The ranking on the 26 tasks counts only submissions that solved at least 7 of them, as in Assignment 4. You get points for the round in which you are ranked:
| Your round | The 26 tasks | Terminal-Bench 2.0 |
|---|---|---|
| 1 | 3 | 2 |
| 2 | 2 | 1 |
| 3 | 1 | 0 |
| 4 or later | 0 | 0 |
For example, if you are ranked in round 1 on the 26 tasks and in round 2 on Terminal-Bench, your bonus is 3 + 1 = 4 percentage points. The bonus counts toward the course's 10% cap on bonus credit and does not change your score on this assignment.