Assignment 2 — Build an agent (10%)

Out Mon Sep 21 · due Sun Oct 4, 11:59pm · individual or pairs · estimated effort 10–12 hours · all five assignments

In Assignment 1 you used an agent. Now you build one.

You will implement a minimal agent runtime around an LLM API. You may use model-provider SDKs, but you may not use an agent framework — no LangChain, LlamaIndex, AutoGen, CrewAI, or equivalent abstraction. By the end of this assignment the mysterious “agent loop” should no longer be mysterious.

Starter code

We provide a skeleton that can call a model but cannot autonomously complete a task:

agent/
    model.py
    tools/
        shell.py
        read_file.py
        write_file.py
    benchmark/
        tasks.json
        evaluator.py
    main.py

The repository link goes here once it is published; it will be announced on the home page and on Canvas before the assignment opens.

Part 1 — The agent loop

while not done:
    context      = construct_context(state)
    response     = call_model(context)
    actions      = parse_actions(response)
    observations = execute(actions)
    state        = update_state(state, actions, observations)

Six lines, and every one of them is a design decision you now own. Your implementation must support:

• model calls
• structured tool calls
• multiple sequential tool calls
• persistent trajectory state
• successful task termination
• maximum-step termination

Part 2 — Tools

Implement at least four tools. Three are required:

• read_file
• write_file
• shell

And at least one of:

• search files
• grep
• Python execution
• HTTP fetch against a provided local service
• documentation lookup
• structured database query

Tool interfaces must have machine-readable schemas. A tool whose contract lives only in your prompt is a tool the model has to guess at, and the guessing shows up in your error rate.

Part 3 — Failure handling

Your runtime must handle all of these:

• invalid tool names
• malformed arguments
• tool exceptions
• timeouts
• model responses containing no useful action
• repeated actions
• context limits

The agent should fail gracefully rather than crash. This is what the hidden tests are aimed at, and it is the part most people leave until last.

Part 4 — Benchmark

We provide a small software-engineering benchmark to start you off — tasks in the shape of:

• Fix the bug causing incorrect eviction behavior.
• Add a --json flag while preserving the existing CLI.
• Find and repair the race condition in the provided program.

You also write your own test cases, and they are part of the deliverable — see Part 6. The provided tasks are a seed, not the whole instrument: a benchmark you did not write is one whose edges you do not know.

Report these numbers. They are this assignment's headline result, and Assignment 3, Assignment 4 and Assignment 5 are all measured against them — so define them precisely now and do not move them afterwards:

• task success rate
• wall-clock time
• model calls per task
• tool calls per task
• input tokens per task
• output tokens per task
• cost per successful task

Report both averages and distributions where appropriate. Agent latency is heavy-tailed and one task stalling in a retry loop hides inside a mean, so a mean on its own is a number that conceals its own most interesting case.

Part 5 — Trace one trajectory

Produce a timeline for one successful task. Something in this shape:

0s          5s         10s        15s        20s
|-----------|-----------|-----------|-----------|

LLM ███████
           read ██
                 LLM ████████
                           shell ███
                               LLM ██████

Annotate model execution, tool execution, idle time, and failures or retries. Then answer one question: what is the critical path? Everything Assignment 3 does is an attempt to shorten it, so a vague answer here is a problem you inherit.

Part 6 — Contribute test cases

Write four problems — one easy, two medium, one hard — that an agent can be asked to solve, each with a reference solution and a scorer. Run all four through the agent you just built; at least two must pass. The class's problems are pooled, and a selection joins the provided tasks as the held-out grading set for Assignments 3, 4 and 5.

Four requirements, and a submission that misses one is not usable:

1. Self-contained. It must run against anyone's agent — no dependency on your repo layout, your tools, your keys, or a service needing an account. Ship any data it needs.
2. Machine-checkable. A scorer that takes the agent's output and exits 0 or 1, and a stated output format. No human judgment in the loop.
3. Definitely answerable. One right answer, or a clearly bounded set, and you know it. A problem is not hard because it is ambiguous — that is a broken problem, and it is the most common failure mode here.
4. On topic. Agents, serving, GPUs, kernels, cost and performance reasoning. A hard Sudoku is not on topic.

Difficulty is measured, not asserted. Run each problem five times against your own agent and report the pass count: easy 5 of 5, medium 2–4 of 5, hard 0–1 of 5. The two mediums should fail for different reasons — say which, in one sentence each. Two problems that break the same way are one problem. A problem whose reference solution is wrong scores nothing for that problem, however good the rest is.

Required experiments

Run at least these two.

Experiment A — Maximum steps

Evaluate maximum trajectory lengths of 4, 8, 16, 32. How does success rate change? How does cost change? The interesting part is where the two curves stop moving together.

Experiment B — Context

Compare full trajectory history against a simple context-truncation policy. Characterize the quality/cost tradeoff. You are measuring, on your own agent, the thing the serving half of this course spends two months on.

Deliverables

1. Your implementation.
2. Benchmark results.
3. Your four problems with reference solutions, scorers, and measured pass counts.
4. One trajectory visualization.
5. results.csv.
6. A report, 4 pages maximum.

Grading

Hidden tests

Your runtime will be run against:

• malformed model output
• failing tools
• very large tool outputs
• repeated tool calls
• tasks exceeding your expected trajectory length

We will not test model intelligence. We will test the runtime. Every item on that list is something Part 3 told you to handle.