Final project

Choose a problem worth solving, build the system needed to study it, and support one clear claim with evidence.

The assignments specify the problem and the evaluation. The project asks you to make both decisions. You may build a tool that fills a practical gap or conduct original research on agents or the systems that run them. The tracks use the same deadlines and rubric dimensions, but the research track has a higher bar for novelty and evidence.

The project is worth 20% of the course grade and is completed individually. Each student submits one proposal, presents one poster and demo, and publishes one blog post and repository.

You do not need to wait for starter code or a release date. Start when you have a problem worth testing. Email the instructor if you want feedback on scope, a compute allocation, or a reader for a draft.

Choose a track

Choose the track that matches the claim you want to make. Name it in the proposal. You may switch later by emailing the instructor.

Product track: build a useful tool

Build an AI system that solves a real problem better than the strongest available alternative. The gap may come from missing context, poor reliability, high cost, slow response, privacy constraints, or a workflow that existing tools do not support.

Suitable directions include:

• Research workflows. Automate experiments, reproduce results, analyze papers, or manage research artifacts.
• Learning. Build a grounded study tool or a system for practicing a skill.
• Personal workflows. Improve a repetitive or frustrating task in your own work or daily life.
• A course extension. Take an assignment substantially beyond its graded requirements. Work you already submitted is the starting point, and the proposal must identify what is new.

Research track: conduct original research

This track has a higher bar. It asks for an original, falsifiable claim that advances what is known about agents or AI systems. A polished implementation is not enough. A reproduction, benchmark, or collection of measurements is not enough by itself either. The work must establish a result that the closest published research does not already provide, explain why the result occurs, and define the conditions under which it holds.

You may work on either side of the course:

• Agent-side research. Study or improve planning, tool use, memory, context management, verification, multi-agent coordination, evaluation, or another part of the agent loop. The unit of analysis is the agent's behavior, capability, reliability, or cost.
• Systems-side research. Study or improve inference, batching, scheduling, caching, routing, quantization, compression, kernels, or another layer that runs the agent or model. The unit of analysis is system performance, efficiency, scalability, or reliability.

On either side, the contribution may take one of four forms:

• Measurement. Reveal a behavior, bottleneck, or tradeoff that prior work does not report, then explain its cause and boundary. Moving an existing benchmark to a different model or machine is insufficient unless the change produces a new conclusion.
• Mechanism. Propose a method, policy, or design and compare it with the strongest published alternative. Isolate the mechanism with an ablation and explain when it helps, when it does not, and why.
• Reproduction and boundary. Reproduce a published result, then identify the workload or condition under which the claim stops holding. Reproduction alone is a starting point rather than a contribution.
• Instrument. Build a benchmark, trace, or measurement harness that enables a previously unanswered question. Validate the instrument, then use it to produce the result.

Find a gap you can test

Before building, determine what already exists and state its limitation in one testable sentence.

For a product project

Run the closest commercial tools, open-source systems, research prototypes, and frontier models on your own task. Pasting the input directly into a frontier model is a baseline, even if it seems too simple. Name the concrete failure you observe. “It cannot access my cluster queue” is a testable gap. “Nothing quite like this exists” is not.

For a research project

Read the closest published work on the agent side or the systems side and identify the precise claim it leaves unsettled. The gap often comes from one of three places. First, nobody measured the behavior on your workload, model, or hardware. Second, a published result uses conditions that should change the answer. Third, two results disagree. “We will try a new idea and see whether it works” is not yet a research question.

Reproduce before you improve. A number measured on somebody else's setup is context, not a baseline. Reproduce the closest result in your setup so that the effect of your change is not confused with a different model, prompt, task set, GPU, software version, or workload. If the published artifact cannot be reproduced, document why and build the closest defensible comparison.

A small gap supported by evidence is stronger than a large gap supported by assertion. Finding out in week one that a tool already solves the problem or that a paper already answers the question costs an afternoon. Finding out in week five costs the project.

Build the evidence

Every project needs four parts:

1. Problem. A concrete problem or question and a demonstrated gap.
2. System. A working tool, mechanism, or experimental harness.
3. Evaluation. Measurements that support the claim and define when it holds.
4. Analysis. An honest account of cost, limitations, failure modes, and threats to validity.

Use the right baselines

If you introduce a mechanism, isolate it with an ablation or another controlled comparison. Report the alternatives and configurations that lose as well as those that win. A negative result is useful evidence. Omitting the strongest comparison is not.

The research evidence bar

A research project must make the result credible beyond the particular run that produced it. State the hypothesis and fix the protocol before the main experiment. Use repeated trials, report variation or uncertainty, isolate the proposed cause, test at least one boundary condition, and preserve the artifacts needed to reproduce the result.

For agent-side research, fix the model version, prompts, tools, task set, and success criteria. Use held-out tasks when the method was tuned on examples, and separate capability from token cost, latency, and stochastic variation. For systems-side research, fix the hardware and software stack, describe the workload, measure under controlled load, and report both central and tail behavior where applicable.

A single benchmark score, a favorable prompt example, or one throughput number is not a research result. The result must survive a strong baseline, repeated measurement, and an attempt to make it fail.

Measure what matters

Choose metrics that match the claim. Agent projects may need success rate, latency, cost, token usage, steps, resource use, and human effort. Serving projects may need throughput, time to first token, per-token latency, tail latency, memory use, cache hit ratio, and prefill work avoided. A latency result must also state the request rate at which it was measured.

Measure one level deeper. Break the top-line result into the components that explain it. “The agent took 60 seconds” says what happened. “Tool calls took 40 seconds, including one 30-second call that could run in parallel” says what to change. Ousterhout's Always Measure One Level Deeper explains why this habit matters.

Fix the workload, task set, and metric before the main experiment. Product projects should record the by-hand workflow before the tool changes it. Research projects should not choose the workload after seeing which one makes the result look strongest.

The goal is a defensible answer, not a favorable answer. A well-explained failure can be a strong project. A speedup caused by a measurement artifact cannot.

Deliverables and deadlines

These are submission deadlines, not start dates. Submit the proposal through Canvas. Present the poster and demo in person. Publish the blog post and repository by opening a pull request to the course blog repository; the deadline is when the pull request is open, not when it is merged.

Proposal (Oct 28, 2%)

Keep the proposal to one page. Name the track and hardware, then cover four items:

1. The problem you will solve or the question you will answer.
2. The closest existing systems or published results, what you tested or reproduced, and the gap that remains.
3. The system or experiment you propose to build.
4. The baselines, workload, and metrics you will use to evaluate it.

A research proposal must also identify whether it is agent-side or systems-side research, state the falsifiable claim or hypothesis, and explain why the planned evidence would meet the higher research bar. The proposal exists to test whether the project is interesting and feasible before most of the work begins. The track is not binding; email the instructor if the work changes direction.

Poster and demo (December, date to be confirmed, 10%)

Present the project at the poster session after classes end. A visitor should be able to identify the problem, the system, and the result quickly. Use at least one figure that remains legible from two meters away. When a live demonstration is appropriate, prepare a reliable version that runs in about two minutes.

Peer evaluation

The poster session includes peer evaluation, and peer scores contribute to the project grade. Each student reviews several other projects. Reviewing is part of the deliverable.

Blog post and repository (Dec 8, 8%)

Write for a general technical audience, using the assignment write-ups as the model. Explain the problem or question, the gap, the system design, the experimental method, the results and comparisons, and the limitations and failure modes. Link to the repository and any public demo. There is no length requirement; keep the post focused.

The repository must let another person reproduce the main result. Include an installation and execution guide, the raw measurements behind each figure, the scripts that generate the figures, and the hardware, software versions, and workload used for every reported number. Label all axes and units, and keep figures legible in print and grayscale.

Include the note required by the AI use policy. State what you delegated, what you kept, where AI clearly helped, and where it misled you. Explain how you detected the problem and what it cost.

What we look for

The four dimensions below are weighted equally. Both tracks use these dimensions, but research projects are held to a higher standard of novelty, experimental control, reproducibility, and engagement with published work because they claim to extend the research record.

A project that does not work can still earn nearly full credit if it explains what you expected, what you measured, and why the two differ. A number you cannot defend loses credit regardless of its sign or size.

Plan for five weeks

1. Week 1. Fix the problem or question. Test the closest systems or reproduce the closest result. Record the baseline.
2. Week 2. Run an end-to-end prototype or a small version of the complete experiment.
3. Weeks 3 and 4. Improve the system and run the controlled experiments.
4. Week 5. Analyze the results and prepare the blog post, repository, poster, and demo.

Obtain the first measurement early. Do not spend four weeks building infrastructure before running the complete path once. If the project needs a GPU, count queue time and reruns as part of the five weeks.

Most projects should fit on a laptop, model API access, or a small number of GPUs. Discuss unusual hardware or substantial compute with the course staff early. Assignment 5 is due Sat Nov 28, the poster session follows, and the project report is due Tue Dec 8, so do not leave either write-up to the final two weeks.

Choose the compute you need

The computing setup page contains access instructions. Choose the simplest resource that can run the experiment, and contact the instructor early if the project needs something not listed here.

Start with the least expensive model and the smallest complete experiment. Cap API requests before a run, stop cloud jobs when they finish, and run a small sweep before launching the full one. A late GPU allocation is not grounds for an extension, but the course staff can help you debug access and scope.