An open-ended opportunity to apply what you learned in this course to a problem you care about. The assignments define the workload and the evaluation for you. Here you choose the problem, build the system, and decide how to evaluate it.
The project is worth 12% of the course grade and can be completed individually or in teams of two or three. Teams submit one blog post and one repository with a short statement of who did what, and everyone on the team gets the same mark unless that statement says otherwise. Email the instructor for feedback on scope, for a compute allocation, or for a reader on a draft.
Three things are due. A short proposal on Oct 28 (2%), a poster and demo at the session held after classes end (8%), and a blog post plus repository on Dec 8 (5%). The proposal is the one that can save you the other five weeks, because it exists to settle whether the problem is interesting and feasible before you build anything.
Build an AI system or tool that solves a problem that existing systems do not solve well. Four directions, weighted equally by the rubric.
These are directions, not project specifications. Finding the right problem is part of the project.
Before building anything, understand what already exists. Identify and test the closest existing systems, including commercial tools, open-source systems, research prototypes, and frontier models (your input pasted straight into one is the baseline most people skip). Then articulate a concrete limitation.
What can your system do that the strongest existing alternative cannot?
The gap does not need to require a new algorithm. It may come from missing data, application-specific context, cost, privacy, latency, reliability, hardware constraints, or integration with a real workflow. Name it in a sentence specific enough to test. "It cannot see my cluster's queue" is a gap. "Nothing quite like this exists" is a statement that you did not look.
A small gap demonstrated convincingly is better than a large gap asserted without evidence. Running the closest existing systems is the cheapest part of the project and the only part that can end it early. Finding out in week one that something already does the job costs an afternoon. Finding out in week five costs the project.
A good project has four components.
Your evaluation should include two baselines. First, the strongest existing system you can find, run on your own case, which is the one that might beat you. Second, how the task is currently done without your system (the by-hand baseline), timed on the same tasks before the tool exists. Take the second one first: it stops being measurable the moment the tool works, and "it used to take forever" is not a number.
The existing system may win. That is a real finding, and it earns more credit than a favourable comparison against nothing. However, not running it at all loses credit.
Choose metrics appropriate for the problem. For agent systems, useful metrics include success rate, latency, cost, token usage, number of steps, resource consumption, and human effort. The five numbers from Assignment 2 cover most of that list, and the instrumentation already exists.
The goal is not to prove that your system wins. The goal is to answer the question you posed convincingly. A well-understood negative result is a good project. A 3× speedup that turns out to be a measurement artifact is not.
Know what a silent error costs. A tool that summarizes papers is wrong cheaply, because you notice. A tool that rewrites your bibliography, resubmits cluster jobs, or edits files in place is wrong expensively, and those errors are the ones you do not see. Classify the tool by the cost of an undetected error, then build the check that catches it before the next feature.
You are the user, so you are also the eval set. That is this project's strength and its trap, because convenient conclusions are easy to reach when you write both the tool and the questions. Write the questions down before you know what the tool turns out to be good at, and keep the numbers that embarrass you.
Three submissions, all through Canvas, plus the poster session itself.
| Due | What | Weight |
|---|---|---|
| Wed Oct 28, 11:59pm | Proposal | 2% |
| December (date to be confirmed) | Poster and demo, including peer evaluation | 8% |
| Tue Dec 8, 11:59pm | Blog post and repository | 5% |
Submit a short proposal describing four things. First, the problem you want to solve. Second, the closest existing systems and their limitations. Third, your proposed approach. Fourth, how you will evaluate it. Name the hardware and the team while you are there.
The main purpose of the proposal is to determine whether the problem is both interesting and feasible before you spend several weeks building it. There is no length requirement, but we encourage you to keep it short and focused.
Present the project at the end-of-semester poster session, held after classes end. Your poster should communicate three things clearly. First, what problem did you solve? Second, what did you build? Third, what did you learn? You should also have a short live demo when appropriate.
Budget real time for it. The poster carries more credit than the blog post and it is read by people with an hour and a room full of other posters. One figure legible from a metre away, a demo that runs in two minutes, and the gap in one sentence is the whole design brief.
The poster session includes peer evaluation. Each student reviews several other projects, and those peer scores contribute to the project grade. Reviewing is part of the deliverable rather than a favour: the people best placed to test a claim that nothing existing does the job are the ones who spent the same semester building the same kind of thing, and they are the ones who will ask "isn't that just X?" out loud. Four other answers to the question you spent five weeks on is also the cheapest calibration available.
Write a blog post in the same form as the assignment write-ups (Assignment 1 sets the pattern). It should be accessible to a general audience and explain six things: the problem and the gap, your system design, the experimental methodology, the results and comparisons, the limitations and failure modes, and a link to your repository or demo. There is no length requirement, but we encourage you to keep it short and focused.
The repository carries as much weight as the prose. It needs a README somebody else can install and reproduce your main result from, the raw measurements behind every figure, the scripts that turn one into the other, and the hardware, software versions, and workload reported for every number you present. Figures should stay legible in print and in grayscale, with labelled axes and units.
Include an AI-use note, per the AI use policy: what you delegated and what you kept, one place AI was clearly better than you, and one place it misled you, how you noticed, and what it cost.
Projects are evaluated along four dimensions, weighted equally.
| Dimension | What is asked | Warning sign |
|---|---|---|
| Problem and motivation | Is there a clear problem? Did you investigate what already exists? Is the gap concrete and convincing? | A vague question, or a problem an existing tool already handles and you never ran |
| Technical execution | Did you build or measure something substantive? Is the implementation technically sound? | Code that only works on your machine, on one input, once |
| Evaluation | Are the baselines and metrics appropriate? Do the experiments actually support the conclusions? | Single-run numbers presented as fact, or a baseline chosen because it loses |
| Reporting | Are the claims clear and supported by evidence? Are limitations and negative results reported honestly? | Figures that look impressive but do not show what the text says they show |
A project that did not work earns nearly full credit, given a clear account of what you expected, what you measured, and why the two disagree. The failed attempts are usually the interesting part. What loses credit is a number you cannot defend.
Every strong systems paper has a "where this does not work" paragraph. Write yours first. If you cannot, you do not yet understand what you measured.
You have approximately five weeks. Scope accordingly. A useful progression is:
Get an end-to-end system running early. Do not spend four weeks building infrastructure before obtaining the first result. A project that produces its first measurement in the first two weeks almost always finishes, and one that spends a month on infrastructure usually does not.
Most projects should fit on a laptop, API access, or a small number of GPUs (the compute resources below are the options). If your project requires unusual hardware or substantial compute, discuss it with the course staff early rather than in the last two weeks.
Two deadlines land six days apart. Assignment 5 is due Wed Dec 2 and this blog post on Tue Dec 8, with the poster session in between, so the last week is tight by construction. Do not leave either write-up to it.
The project is not a competition to produce the largest speedup or the most impressive demo.
Find an interesting problem and need, then build a tool that solves it. Ideally this is a tool you will keep using in your daily life and work after the semester ends.
Setup instructions for all of it live on the computing setup page, which covers the assignments and this project alike. What follows is how to choose for project work. If you need hardware beyond what is listed, talk to the instructor early. Sometimes it can be arranged, but not in the last two weeks.
This provides both CPU and GPU (RTX PRO 6000 Blackwell) resources for running your project. Note that the CPU partition has a max time of 3 days and GPU partition has a max time of 2 days. This HPC runs on AWS and please stop your jobs when you are done to avoid unnecessary charges. The HPC is the right choice for most projects, Note that we do not enforce how many GPU jobs you submit, but the total GPU pool is limited, so please be considerate of other students and please start your jobs early. We will not grant extensions for late jobs because you cannot allocate a GPU (but certainly send us an email if you need help).
This course is partially sponsored by AWS and each student receives cloud computing credit. The request form is only available in the first week.
FASRC provides access to a large number of CPU and GPU nodes for research and teaching. It has more powerful GPUs such as H100 and H200, however, they often have a queue and students are not automatically granted access. Please request access as early as possible if you have a need for it.
CloudLab provides bare-metal machines for research and teaching, free of charge, with full control over the software stack. This is the right choice when your project needs to touch the kernel, the network, or the storage layer, or when you need reproducible isolation that a shared HPC cannot give you.
Please talk to the instructor or TA if you have any questions about the available options.