CS2680 Modern AI Systems: Agents and System Optimizations
HPC: running an agent in an interactive job

The HPC

The easiest path is an interactive Slurm job started from an Open OnDemand terminal — not work done on the login node. One srun command gets you a shell on a compute node; you then start claude in it and work normally.

If you have never used an HPC before, read how the HPC works first — it is four paragraphs, and it explains why the one-line answer above is the one-line answer. If you have, the quick start is all you need.

Access: partly TBD. The login host is academ-acade-iL73aitWT6xF-c83014867702e61e.elb.us-east-1.amazonaws.com — see the SSH config below, which gives it a short name. Still to be announced: the OnDemand portal URL, and how you get an account, register your SSH key, and get added to the CS2680 group. Those will be posted here and on the home page.

Quick start

Once, ever — from an OnDemand terminal on the login node:

curl -fsSL https://claude.ai/install.sh | bash # installs to ~/.local/bin echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc source ~/.bashrc claude # sign in, then /exit

Then every time you sit down to work:

srun -p general -c 4 --mem=16G -t 4:00:00 --pty bash # get a compute node claude # work here

And when you need a GPU instead (Assignment 4 onward):

srun -p gpu-cs2680 --gres=gpu:rtxproserver6000:1 -c 4 --mem=16G -t 4:00:00 --pty bash

exit ends the job and releases the node. That is the whole workflow; the rest of this page is what to do when it does not go like that.

How the HPC works

There are three machines in this story. Your laptop, where you type. The login node, which is what you reach when you open an OnDemand terminal or SSH in — one small shared machine that everyone in the course lands on at once. And the compute nodes, which are the actual hardware: many CPUs, lots of memory, and in one partition, a GPU.

You are not allowed to just run things on a compute node. Slurm — the scheduler — owns them, and you ask it for a lease: give me 4 CPUs and 16 GB for 4 hours. When hardware is free it hands you a node and starts your shell on it. That lease is a job, and everything you run inside it is charged against it. When the time runs out, or you type exit, the lease ends.

Do not run an agent on the login node. The login node is capped at 15 minutes of CPU time and roughly 2 GB of memory. An agent session will be killed there, usually in the middle of something, and often without a clear message. The login node is for submitting jobs, editing files, and moving data. Everything with real work in it belongs in a job.

The one thing that makes this pleasant is that /home is shared. The login node and every compute node see the same home directory, so anything you installed once is already there when a job starts — claude in ~/.local/bin, or grok in ~/.grok/bin. Nothing to re-install per job, nothing to copy onto the node, and the login you did in week one still applies, because the credentials Claude Code writes live in your home directory too.

Two kinds of job, and you will mostly want the first. An interactive job (srun --pty) gives you a shell and you work in it, which is what an agent session is. A batch job (sbatch) runs a script without you and writes its output to a file, which is what you want for an overnight sweep.

First-time setup

Six steps, once. Do them on the login node — they are all small, and installing software is exactly what the login node is for.

  1. Get an account, and get into the CS2680 group. Access instructions are TBD and will be announced; the group membership is what lets you submit to the gpu-cs2680 partition.
  2. Open a terminal. In the OnDemand portal, the shell is under the Clusters menu; it opens a terminal on the login node in a browser tab. Plain SSH to the login host works the same way and is nicer if you already live in a terminal — the SSH config below reduces it to ssh ood.
  3. Install Claude Code into your home directory:
    curl -fsSL https://claude.ai/install.sh | bash
    This is the native installer, it puts the binary in ~/.local/bin, and it keeps itself updated. Because /home is shared, you are also installing it on every compute node at the same time. If you use grok instead, its installer lands in ~/.grok/bin and everything on this page applies unchanged.
  4. Put it on your PATH. Many HPC shells do not include ~/.local/bin:
    echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc source ~/.bashrc claude --version
    ~/.bashrc rather than ~/.bash_profile, because a Slurm job may start either a login or a non-login shell depending on how it was launched, and only ~/.bashrc is read in both cases — provided your ~/.bash_profile sources it, which is the default on most systems and worth confirming with grep bashrc ~/.bash_profile.
  5. Log in. Run claude and follow the prompts. In a browser-based terminal or over SSH you will get a code to paste back rather than a browser that returns to the terminal — that is expected, and the Claude Code page walks through it. Confirm with /status, then /exit.
  6. Add the course telemetry block to ~/.claude/settings.json on this machine, per the Claude Code page. That file is per-machine, and your HPC home is a different machine from your laptop as far as it is concerned.
Sessions you run here are sessions you submit. Claude Code keeps its transcripts under ~/.claude/projects/ on whichever machine ran it, so the HPC has its own set. When you archive your session files, include the HPC's — and scan them for credentials like any other.

Interactive jobs: the everyday workflow

From an OnDemand terminal (or over SSH):

srun -p general -c 4 --mem=16G -t 4:00:00 --pty bash

Wait for the node to come up. The first job of the day can take a few minutes, because idle nodes are powered down — sinfo shows them as idle~, and Slurm has to boot one before your shell appears. Nothing is wrong; squeue -u $USER in another terminal shows the job pending with a reason.

When the prompt comes back you are on the compute node — hostname will show a node name rather than the login host. Then:

claude # or: grok

That is the whole student workflow: allocate a shell on a compute node, run the agent there. Work normally, and exit when you are done, which ends the job and returns the node.

What the flags mean

Flag What it does How to choose it
-p general Partition — which pool of nodes to run on. general for anything that is not GPU work. See GPUs for the other one.
-c 4 CPU cores. 4 is plenty for an agent. More cores means a longer wait for a node with that many free.
--mem=16G Memory for the whole job. Exceed it and the job is killed. 16 GB is comfortable for agent work. Raise it if you are loading data or model weights.
-t 4:00:00 Wall-clock limit, HH:MM:SS. At the limit the job ends, mid-command if need be. Ask for a working session, not a week. Check the partition's ceiling with sinfo -p general -o "%P %l".
--pty bash Attaches a terminal and runs a shell in it — this is what makes the job interactive. Always, for this workflow. Swap bash for tmux new -s agent to make it survive a closed tab.

A CPU partition is the right default

A coding agent talks to an API over the network. The model runs in a datacenter that is not this one, so the agent needs a few cores to run tools, compile things, and hold a repository in page cache — and no GPU at all. Asking for a GPU you will not use means waiting in a longer queue for hardware you then leave idle, which is the one thing a shared HPC punishes.

Reach for a GPU when the assignment is about serving a model yourself: Assignment 4 onward.

GPUs (CS2680 only)

The course has its own GPU partition:

srun -p gpu-cs2680 --gres=gpu:rtxproserver6000:1 -c 4 --mem=16G -t 4:00:00 --pty bash

Each node in it is 1× RTX PRO Server 6000, 8 CPUs, and roughly 61 GB of RAM. So the request above is about half a node, and asking for more than a node has — nine cores, 64 GB, two GPUs — produces a job that waits forever rather than an error. Only members of the CS2680 group can submit here; an invalid-partition or invalid-account message means that is what to fix first.

Check the GPU is really yours, first thing inside the job:

nvidia-smi echo $CUDA_VISIBLE_DEVICES

nvidia-smi should list one card with almost no memory in use, and CUDA_VISIBLE_DEVICES should name the one device Slurm gave you. If nvidia-smi reports no devices, you are not in the job you think you are — check hostname and squeue -u $USER.

Serving a model here (Assignments 4 and 5)

Two things will bite you, and both are about disk rather than the GPU.

  • Model weights are large and your home directory is not. A 7–8B model in bf16 is roughly 15 GB, and Hugging Face downloads into ~/.cache/huggingface by default, so the third model you try is where the quota error appears. Point the cache somewhere with room before you start — export HF_HOME=/path/with/space/$USER/hf in your ~/.bashrc — and delete the models you were only curious about. TBD: the scratch filesystem to use for this, once HPC access is finalized.
  • Install the engine into an environment, not the system. A virtualenv or a conda environment under your home works on every node, for the same shared-/home reason. Create it once, activate it in the job.

Ask for more of the node when you are serving rather than chatting — -c 8 --mem=48G leaves the weights room to load — and remember that a whole-node request only starts when a whole node is free. And kill the server when you stop measuring: a vLLM process holding 90 GB of VRAM in a job you have wandered away from is invisible to you and obvious to everyone waiting.

A GPU you are not using is a GPU nobody is using. There is one card per node and a whole class sharing them, so bound the job with -t, and exit when you stop working rather than holding a four-hour allocation to read a paper. This bites hardest in the last week of the semester, which is exactly when you will want a node at short notice.

One command from your laptop: ssh ood1

Once the workflow above is familiar, you can collapse it into a single command. SSH will run srun for you on connect, so ssh ood1 takes you from your laptop to a shell on a compute node with nothing typed in between. Put this in ~/.ssh/config on your own machine (C:\Users\you\.ssh\config on Windows), with your username and key path:

Host ood ood1 ood2 Hostname academ-acade-iL73aitWT6xF-c83014867702e61e.elb.us-east-1.amazonaws.com User YOUR_USERNAME IdentityFile ~/.ssh/id_ed25519_seas IdentitiesOnly yes ForwardAgent yes ServerAliveInterval 30 ServerAliveCountMax 20 Host ood1 RequestTTY force RemoteCommand /opt/slurm/bin/srun --partition=general --nodes=1 --ntasks=1 --cpus-per-task=4 --mem=16G --time=6:00:00 --job-name=cpu --chdir=$HOME --pty bash -l Host ood2 RequestTTY force RemoteCommand /opt/slurm/bin/srun --partition=gpu-cs2680 --nodes=1 --ntasks=1 --cpus-per-task=4 --mem=16G --gres=gpu:rtxproserver6000:1 --time=6:00:00 --job-name=gpu --chdir=$HOME --pty bash -l

That gives you three names for the same machine:

  • ssh ood — a plain shell on the login node. This is where you install things, move files, and check the queue.
  • ssh ood1 — a CPU compute node: 4 cores, 16 GB, 6 hours. Agent work goes here.
  • ssh ood2 — a GPU compute node: the same, plus one RTX PRO Server 6000.

The srun flags are the long-form spellings of the ones in the flag table — --cpus-per-task=4 is -c 4, --time=6:00:00 is -t 6:00:00 — so edit them the same way. Of the SSH directives, four earn their place: RequestTTY force and RemoteCommand are what replace your login shell with the job; ForwardAgent yes lends the node your laptop's key so git push works from inside the job without a key ever living on the HPC; and the two ServerAlive lines keep an idle connection from being dropped by something in the middle, which is what usually kills a long agent session.

Generate the key first if you do not have one, and register it per the access instructions above:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_seas
Every ssh ood1 is a new job. Two terminals means two allocations, and a reconnect after your wifi drops does not put you back in the old one — it queues for another node while the first job sits there holding hardware until its six hours are up. Check with squeue -u $USER (the --job-name in the config is why they show up as cpu and gpu), scancel the strays, and use tmux inside the job if you want something to survive a lost connection. To get back into a job that is still running, ssh ood and then srun --jobid=<jobid> --pty bash.

One more consequence of RemoteCommand: on ood1 and ood2 there is no plain shell to talk to, so scp, rsync, sftp, and editors that drive SSH themselves — VS Code Remote-SSH included — will not work through those two names. Use ood for all of them: rsync -avz ./data/ ood:~/cs2680/data/.

Surviving a closed tab: wrap it in tmux

Recommended for this class. An srun --pty shell belongs to the terminal that started it, so a closed browser tab, a laptop lid, or hotel wifi takes your agent with it — mid-edit, and the job dies too. Start the job inside tmux instead:

srun -p general -c 4 --mem=16G -t 4:00:00 --pty tmux new -s agent # inside tmux: claude

Now the agent is a process inside a tmux session on the compute node, and losing your terminal only detaches it. To get back:

squeue -u $USER # find the node your job is on # if the allocation is still alive, attach from another srun/ssh, or just tmux attach -t agent # if you are already on that node

Three things to keep straight:

  • The session lives on one specific node. You have to be on that node to attach, which is what squeue -u $USER tells you.
  • tmux does not extend the allocation. When the walltime in -t runs out, Slurm ends the job and the tmux session with it. Nothing survives the lease.
  • Detach on purpose with Ctrl-b d. That leaves everything running. Ctrl-b c opens another window in the same session, which is how you watch a log while the agent works.

Long and unattended runs

Interactive jobs are for working. For an evaluation sweep — the same agent over fifty tasks, the kind of thing Assignment 3 onward asks for — you do not want to be present at all. Write a script:

#!/bin/bash #SBATCH -p general #SBATCH -c 4 #SBATCH --mem=16G #SBATCH -t 8:00:00 #SBATCH -J agent-eval #SBATCH -o logs/eval-%j.out cd "$HOME/cs2680/a3" python run_eval.py --tasks tasks.jsonl --out results/

Submit it with sbatch run_eval.sh, which prints a job id and returns immediately. squeue -u $USER tracks it, %j in the output path expands to the job id so concurrent runs do not overwrite each other, and everything the script prints lands in that file. Create the logs/ directory first — Slurm will not, and a job whose output file cannot be opened fails instantly for a reason that is hard to see.

This is also the honest way to run the agent itself unattended: claude -p "…" is headless and takes the same task from a script that you would have typed, which is exactly the interface Assignment 1 built a console around. It needs no browser login inside the job, because the credentials from your one-time claude login are sitting in your shared home directory.

Check the partition's ceiling before asking for a long job. sinfo -p general -o "%P %l %D %c %m %G" prints the partition's time limit alongside its node count, cores, memory and GPUs. A -t above the limit is rejected at submission; a job sized above any node simply never starts, which looks identical to a busy HPC.

Getting your code and data there

  • Use git for code. Clone on the login node, commit and push from wherever you worked. This is also how your laptop and the HPC stop disagreeing about which version you measured.
  • Use rsync for everything else. From your laptop:
    rsync -avz ./data/ ood:~/cs2680/data/ # up rsync -avz ood:~/cs2680/results/ ./results/ # down
    — ood being the login-node alias from the SSH config; without it, spell out you@ and the full login hostname. rsync over scp because it resumes and skips what has not changed, which matters the third time you copy a results directory.
  • OnDemand has a file browser for the one-off case — dragging a CSV in, pulling a plot out — without a terminal at all.
  • Keep results out of the node's local disk. Write into your home directory or a shared filesystem. Anything a job leaves in /tmp on a compute node is not yours after the job ends.

Command reference

Command What it tells you
squeue -u $USER Your jobs: id, partition, state (PD pending, R running), time used, and the node. The reason column explains a pending job.
sinfo -p general Node states in a partition. idle free, idle~ free but powered down, alloc taken, mix partly taken, down/drain unavailable.
sinfo -p gpu-cs2680 -o "%P %l %D %c %m %G" What you are allowed to ask for: time limit, node count, cores, memory, GPUs. Run this before inventing flags.
scancel <jobid> Ends a job. scancel -u $USER ends all of yours — useful after a lost terminal.
sacct -j <jobid> --format=JobID,State,Elapsed,MaxRSS,ReqMem,ExitCode The post-mortem for a finished job. TIMEOUT means it hit -t; OUT_OF_MEMORY, or a MaxRSS at your ReqMem, means --mem.
scontrol show job <jobid> Everything Slurm knows about a job that is still queued or running, including why it is waiting.
hostname Whether you are on the login node or a compute node. Worth checking when something gets killed.
nvidia-smi Inside a GPU job: the card, its memory, and what is using it.

Troubleshooting

Symptom What it is, and what to do
Nothing happens for minutes after srun. Normal for the first job — a powered-down node is booting. squeue -u $USER from another terminal shows it pending with a reason.
The job never starts. You asked for more than a node has, or the partition is full. Compare your flags against sinfo -p <partition> -o "%P %l %D %c %m %G" and bring -c, --mem, -t and --gres inside it.
Your session was killed with no message. Almost always the login node's 15-minute CPU cap — check hostname. Otherwise the job hit its -t walltime or its --mem; sacct -j <jobid> distinguishes them.
claude: command not found inside a job. PATH is set in ~/.bash_profile, which a non-login job shell never reads. Move the export PATH line into ~/.bashrc, or run it as ~/.local/bin/claude.
Claude Code asks you to log in again on the HPC. Expected the first time: the credentials live in your home directory, not your laptop. Over SSH or a browser terminal you get a code to paste back rather than a browser redirect — see logging in.
Invalid partition, or an account/association error. You are not in the CS2680 group yet, or you typed the partition name wrong. Only gpu-cs2680 is group-restricted.
nvidia-smi reports no devices in a GPU job. You are on the login node, or the job was allocated without --gres. Check hostname and re-submit with the --gres flag exactly as written above.
scp, rsync or VS Code Remote-SSH fails against ood1 / ood2. Those names carry a RemoteCommand, so there is no plain shell for them to talk to. Use the ood alias, which is the login node. See the SSH config.
You have three jobs and you only wanted one. Every ssh ood1 allocates a fresh job, and a reconnect does not reuse the old one. squeue -u $USER, then scancel the strays; use srun --jobid=<jobid> --pty bash from the login node to re-enter one that is still alive.
A job you forgot about is still running. squeue -u $USER to find it, scancel <jobid> to release it. Do this before asking why nothing will start.
Out of disk, or a quota error while downloading a model. Your home directory filled up, usually with ~/.cache/huggingface. See serving a model here.
Something else. Bring the exact command, the job id, and the error text to office hours or the course forum. A job id is enough for anyone to look up what actually happened.

Sharing the HPC

A small number of nodes and a whole class. Four habits, and the last one is the one people forget:

  • Nothing heavy on the login node. It is shared by everyone at once, and the caps are there because someone always tries.
  • Ask for what you need, not for the node. Over-asking makes your own job wait longer and keeps hardware from someone with a right-sized request.
  • Bound every job with -t, and prefer a batch job over an interactive one you will not be watching.
  • exit when you stop working. An idle allocation is invisible to you and expensive for everyone else, and the queue is longest in the week you most need it.