CS2680 Modern AI Systems: Agents and System Optimizations
Fall 2026

Announcement

  • Aug 20, 2026: Please request your license for Claude Code ASAP. It will be used for your first assignment.
  • Aug 20, 2026: Please register an AWS account and add your name, email and account ID to the course sheet. See AWS setup.
  • Aug 17, 2026: Welcome to CS2680 Modern AI Systems: Agents and System Optimizations! Class meets Monday and Wednesday; our first meeting is Wed Sep 2, 2026.

Overview

This course follows one AI system from the agent loop down to the GPU. You first use an agent, replace its framework with a loop you understand, measure it, and optimize it while the model is still a rented API. Then you serve an open-weight model yourself and take responsibility for the layers underneath: GPU kernels, batching and scheduling, KV-cache and prefix reuse, routing, quantization, and speculative decoding. We finish by reversing the question and asking what LLMs can do for systems research. At every layer, the method is the same: define success, account for every token and second, find the binding constraint, and measure what changed.

Prerequisites: At least one of CS61, CS1610, CS2620. Comfort with Python and PyTorch is assumed; prior exposure to machine learning (e.g. CS1810) is recommended but not required.
Textbook: No textbook is required. Background reading: Computer Systems: Operating Systems: Three Easy Pieces
Large Language Models: Ultra-Scale Playbook, Foundations of Large Language Models
Deep Learning: The Little Book of Deep Learning
Class time and location: Monday and Wednesday, 11:15am – 12:30pm, SEC LL2.221
Format: The course runs in two parts.
  • Part I — Introduction to LLMs and Agents. Course overview and modern ML from a systems view across the first two meetings, then agents from a user's perspective and two meetings on designing one — the loop, tools, and context, then the specification, the verifier, and the evaluation harness. Two hands-on assignments: use an agent, then design one. A further self-study track on working with agents from the outside sits under optional content.
  • Part II — Systems for LLMs. The bulk of the course: GPU architecture and Triton kernel programming, attention kernels, and the serving stack — paging, batching and scheduling, routing, a two-meeting block on the KV cache, prefix caching, quantization, speculative decoding, and a two-lecture block on serving agents specifically, plus three guest lectures. Three of the four student sharing sessions land here — Assignment 3's on Oct 28 and Assignment 5's across the last two meetings. Assignment 0's falls in Part I. The training side of the stack is not lectured on and is listed under optional content — parallelism, fine-tuning, and data pipelines, as is the LLMs-for-systems material that used to run as a third part.
Lectures carry the fundamentals; the research-heavy topics run as student-led paper discussions, and four sessions are given over to student sharing after three of the six assignments — both marked Students in the schedule. Students are expected to actively participate. See the paper discussion page for what that involves. The course ends the way it ran: the last two meetings are the final student sharing session, where you show the room the full-stack system you built and optimized across the assignment sequence.
What this course expects: This course will not teach you every piece of knowledge you need for the assignments. It focuses instead on the skills to learn with, and points you in directions worth going. Expect to spend a significant amount of time learning and experimenting on your own after class — if you want an A, plan on it.
Grading:
  • Assignment 0 (use an agent): 10%
  • Assignment 1 (design an agent): 10%
  • Assignment 2 (contribute problems): 8%
  • Assignment 3 (optimize the agent): 16%
  • Assignment 4 (serve your own agent): 10%
  • Assignment 5 (optimize the full stack): 16%
  • Final Project: 16%
  • Paper Presentation: 10%
  • Class Participation: 4%
  • Bonus: up to 10% (course feedback 2%; problems current AI cannot solve, 2% each)
See policy page for details.
Course website: https://cs2680.com

Staff

Juncheng Yang
Juncheng Yang
Instructor
Yiyu Liu
Yiyu Liu
TA
Yunjia Zheng
Yunjia Zheng
TA
Office hours:
  • Instructor: Appointment only
  • TAs: TBD

Schedule

Tentative and subject to change. Rows with Students as the speaker are student-led: either a paper discussion (see the paper discussion page) or an Assignment sharing session — the Topic cell says which. Readings marked required should be read before class; everything else is optional depth.

Week Date Speaker Topic Course Materials Notes
Part I — Introduction to LLMs and Agents
1 2026-09-02 Juncheng Yang Course overview and logistics Assignment 0 out (use an agent); presentation sign-up opens
2 2026-09-07 No class — Labor Day
2 2026-09-09 Juncheng Yang Modern ML basics: transformers
3 2026-09-14 Juncheng Yang Agents from a user’s perspective
3 2026-09-16 Juncheng Yang Agents from a designer’s perspective I Assignment 0 due Sun Sep 20, 11:59pm
4 2026-09-21 Students Student sharing I What you learned from using an agent (Assignment 0): surprises, failures, lessons, insights, new tools/skills/plugins Assignment 1 out (design an agent)
4 2026-09-23 Juncheng Yang Agents from a designer’s perspective II
Part II — Systems for Agents
5 2026-09-28 Juncheng Yang GPU programming for MLSys: architecture, memory, and kernels
5 2026-09-30 Juncheng Yang Efficient LLM computing: GPU kernels Assignment 1 due Sun Oct 4, 11:59pm
6 2026-10-05 Juncheng Yang Efficient LLM serving: paging, batching, and scheduling I Assignment 3 out (optimize the agent); Assignment 2 out (contribute problems)
6 2026-10-07 Students Efficient LLM serving: batching and scheduling II Assignment 2 due Sun Oct 11, 11:59pm
7 2026-10-12 No class — Indigenous Peoples’ Day
7 2026-10-14 Students Efficient LLM serving: routing and load balancing
8 2026-10-19 Juncheng Yang Efficient LLM serving: KV-cache optimization I
8 2026-10-21 Students Efficient LLM serving: KV-cache optimization II Assignment 3 due Sun Oct 25, 11:59pm
9 2026-10-26 Guest Speaker TBD Assignment 4 out (serve your own agent)
9 2026-10-28 Students Student sharing II What you optimized and what it bought (Assignment 3) Project proposal due
10 2026-11-02 Guest Speaker TBD
10 2026-11-04 Juncheng Yang Efficient LLM serving: prefix cache
11 2026-11-09 Juncheng Yang Efficient LLM serving: pruning and quantization Assignment 4 due Tues Nov 10, 11:59pm
11 2026-11-11 Students Efficient LLM serving: speculative decoding Assignment 5 out (optimize the full stack)
12 2026-11-16 Guest Speaker TBD
12 2026-11-18 Juncheng Yang Efficient agent serving systems I
13 2026-11-23 Students Efficient agent serving systems II
13 2026-11-25 No class — Thanksgiving recess
14 2026-11-30 Students Student sharing III What you optimized and what it cost (Assignment 5)
14 2026-12-02 Students Student sharing IV What you optimized and what it cost (Assignment 5) Assignment 5 due Dec 02, 11:59pm
14 (TBD) Students Final project poster and demo Time to show case your fun projects to the world! Final project due Dec 08, 11:59pm; peer reviews due Dec 07, 11:59pm

Optional Content

The topics below are not covered in lecture. None of them has a class meeting or a paper discussion, and nothing on the schedule assumes you worked through them.

Working with agents in practice. Specification, cheap verification, and when a second agent earns its bill. It was on the schedule and came back off it; the lecture notes are written and stay up, so this is a complete meeting you can work through on your own. Assignments 0 and 1 are the place the material actually gets used. The readings page collects the papers with a note on each.

LLMs for systems. Turning the relationship around: models that write, optimize, and evaluate systems code, and the harnesses that decide whether the answer was any good. It ran as Part III on an earlier version of this schedule and came off it when the semester ran out of meetings — the readings are complete, the second meeting's lecture notes are written and stay up, and the material is the premise behind Assignment 5 and a good deal of what a final project tool has to do. The readings page collects the papers with a note on each.

The training side of the stack. How a model gets built, rather than how it gets served. Work through it if you want the background on distributed training, fine-tuning, or checkpointing. The Ultra-Scale Playbook is the best single entry point; the papers below go deeper, and the readings page collects them in one place with more on each.

Topic Materials
Working with agents: specification, verification, and multi-agent workflows lecture notes; SWE-agent, Reflexion, AutoGen, KernelBench
LLMs for systems research I: agents that write and optimize systems code SWE-agent, KernelBench; then LLMs for compiler optimization, OpenHands, SWE-bench Multimodal, BountyBench, measure one level deeper
LLMs for systems research II: automating the research loop lecture notes; MLGym, AI Scientist-v2; then automated statistical model discovery, MLE-bench, SWE-bench, OSWorld
Distributed training and fine-tuning I: parallelism and ZeRO start with Megatron-LM, ZeRO; then FSDP, GPipe, PipeDream, Alpa, LoRA, QLoRA, TorchTitan
Distributed training and fine-tuning II: memory, failures, and scale start with Llama 3 (§3, infrastructure); then MegaScale, DeepSeek-V3, ByteCheckpoint, FP8-LM, ZeRO-Infinity, sublinear memory
Data pipelines and checkpointing Data-Juicer, Dolma, DataComp-LM, CheckFreq, GEMINI, 3FS

The semester closes with a final project worth 12%: build a tool that does not already exist — for your research, your studies, your week, or as an extension of one of the assignments — solving a problem no existing system solves. Try the three closest things that exist, name what each cannot do, then measure your own tool against them and say plainly where it does not work. Announced Oct 26, one-page proposal due Oct 28, report and repository due Dec 8, and a poster and demo at the December session. Half the project's credit is scored by four classmates at that session, and you review four projects yourself. It replaces the cache competition earlier versions of this schedule ran in the same weeks.

Compute Resources

Setup instructions for every environment the course uses are on the computing setup page. Compute for the assignments and the final project comes from:

  • Claude Code — the agent for all six assignments, on an account Harvard FAS provides
  • The course HPC — interactive Slurm jobs, and the gpu-cs2680 GPU partition
  • AWS — where Assignment 0's console is deployed

Sponsors

We are grateful for the support of our sponsors.