The HPC cluster

Harvard FAS ATG provides a high-performance computing (HPC) cluster for this course. It has 72 CPU nodes and 50 GPU nodes. The cluster is managed by Slurm. The cluster is accessible via the Open OnDemand portal. ATG publishes the OnDemand user documentation for it.

We will figure out how to add you to the cluster. The login host is academ-acade-iL73aitWT6xF-c83014867702e61e.elb.us-east-1.amazonaws.com. The SSH config below gives it a short name. Still to be announced: how you get an account and get added to the CS2680 group. Those will be posted here and on the home page. Registering an SSH key is not announced, because you do it yourself: connect through the OnDemand portal once, and append your public key to ~/.ssh/authorized_keys from the browser terminal, as step 3 of first-time setup describes.

Quick start

Once, ever, from an OnDemand terminal on the login node:

Then every time you sit down to work:

And when you need a GPU instead (Assignment 4 onward):

exit ends the job and releases the node. That is the whole workflow; the rest of this page is what to do when it does not go like that.

How Slurm works

There are three machines in this story. Your laptop, where you type. The login node, which is what you reach when you open an OnDemand terminal or SSH in, one small shared machine that everyone in the course lands on at once. And the compute nodes, which are the actual hardware: many CPUs, lots of memory, and in one partition, a GPU.

You are not allowed to just run things on a compute node. Slurm, the scheduler, owns them, and you ask it for a lease: give me 4 CPUs and 16 GB for 2 hours. When hardware is free it hands you a node and starts your shell on it. That lease is a job, and everything you run inside it is charged against it. When the time runs out, or you type exit, the lease ends.

The one thing that makes this pleasant is that /home is shared. The login node and every compute node see the same home directory, so anything you installed once is already there when a job starts — claude in ~/.local/bin. Nothing to re-install per job, nothing to copy onto the node, and the login you did in week one still applies, because the credentials Claude Code writes live in your home directory too.

There are two types of job. An interactive job (srun --pty) gives you a shell and you work in it, which is what an agent session is. A batch job (sbatch) runs a script without you and writes its output to a file, which is what you want for an overnight sweep.

First-time setup

Seven steps, once. Do them on the login node, because they are all small and installing software is exactly what the login node is for.

1. Get an account, and get into the CS2680 group. Access instructions are TBD and will be announced; the group membership is what lets you submit to the gpu-cs2680 partition.
2. Open a terminal. In the OnDemand portal, the shell is under the Clusters menu; it opens a terminal on the login node in a browser tab. Plain SSH to the login host works the same way and is nicer if you already live in a terminal — the SSH config below reduces it to ssh cs2680.
3. Register your SSH key. There is no upload form for this, so you install the key yourself through the OnDemand terminal you just opened, which is the one way onto the cluster that does not need a key already. On your laptop, generate a key if you do not have one, then print the public half:
   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 # skip if that file already exists cat ~/.ssh/id_ed25519.pub
   Copy the single line that prints. Note that the file to copy is id_ed25519.pub; the one without .pub is the private key, and it never leaves your laptop. Back in the OnDemand terminal, append that line to ~/.ssh/authorized_keys:
   mkdir -p ~/.ssh && chmod 700 ~/.ssh echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... you@laptop' >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys
   Paste your own key between the quotes, in place of the example, and keep it on one line. Pasting into the browser terminal is Ctrl+Shift+V rather than Ctrl+V on Linux and Windows, and Cmd+V on a Mac. The two chmod lines are not optional: sshd ignores a key file that the group or the world can write, which is the usual reason a newly added key is refused without explanation. While you are in that terminal, run whoami and write the username down, because the SSH config below needs it. Then check the key from your laptop:
   ssh YOUR_USERNAME@academ-acade-iL73aitWT6xF-c83014867702e61e.elb.us-east-1.amazonaws.com
   A shell prompt on the login node means the key is registered. If you are asked for a password instead, or told Permission denied (publickey), re-open the OnDemand terminal and compare cat ~/.ssh/authorized_keys against cat ~/.ssh/id_ed25519.pub on your laptop, character for character. A key broken across two lines by the paste is the most common failure.
4. Install Claude Code into your home directory:
   curl -fsSL https://claude.ai/install.sh | bash
   This is the native installer, it puts the binary in ~/.local/bin, and it keeps itself updated. Because /home is shared, you are also installing it on every compute node at the same time.
5. Put it on your PATH:
   echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc source ~/.bashrc claude --version
6. Log in. Run claude and follow the prompts. In a browser-based terminal or over SSH you will get a code to paste back rather than a browser that returns to the terminal. That is expected, and the Claude Code page walks through it. Confirm with /status, then /exit.
7. Add the course telemetry block to ~/.claude/settings.json on this machine, per the Claude Code page. That file is per-machine, and your HPC home is a different machine from your laptop as far as it is concerned.

Interactive jobs: the everyday workflow

From an OnDemand terminal (or over SSH):

Wait for the node to come up. The first job of the day can take a few minutes, because idle nodes are powered down. sinfo shows them as idle~, and Slurm has to boot one before your shell appears. Nothing is wrong; you can type squeue -u $USER in another terminal to see the job pending reason.

When the prompt comes back you are on the compute node. Then:

That is the whole student workflow: allocate a shell on a compute node, run the agent there. Work normally, and exit when you are done, which ends the job and returns the node.

GPUs (Assignment 4, 5 and project)

The course has its own GPU partition:

Each node in it holds one RTX PRO 6000 Blackwell.

Use gpu-cs2680 and nothing else. The cluster carries other GPU partitions, and because only gpu-cs2680 is group-restricted, some of them will accept a job from you. However, using one for course work requires pre-approval from a TF. Ask first, and use gpu-cs2680 in the meantime.

Check the GPU is really yours, first thing inside the job:

nvidia-smi should list one card with almost no memory in use, and CUDA_VISIBLE_DEVICES should name the one device Slurm gave you. If nvidia-smi reports no devices, you are not using the correct partition.

One command from your laptop: ssh cs2680

Once the workflow above is familiar, you can collapse it into a single command. SSH will run srun for you on connect, so ssh cs2680 takes you from your laptop to a shell on a compute node with nothing typed in between. Put this in ~/.ssh/config on your own machine (C:\Users\you\.ssh\config on Windows), with your username and key path:

That gives you three names for the same machine:

• ssh cs2680. A plain shell on the login node. This is where you install things, move files, and check the queue.
• ssh cs2680-cpu. A CPU compute node: 4 cores, 16 GB, 2 hours. Agent work goes here.
• ssh cs2680-gpu. A GPU compute node: the same, plus one RTX PRO 6000 Blackwell.

All three names authenticate with the key you installed in step 3 of first-time setup. If you skipped that step, do it now, because none of these aliases will connect until the public key is in ~/.ssh/authorized_keys on the cluster. VS Code’s Remote-SSH accepts the same three names and behaves the same way a terminal does. Remember to scancel JOB_ID once you are finished with a job, because otherwise it holds its node for the whole allocation.

Surviving a closed tab: wrap it in tmux

Recommended for this class. An srun --pty shell belongs to the terminal that started it, so a closed browser tab, a laptop lid, or hotel wifi takes your agent with it, mid-edit, and the job dies too. Start the job inside tmux instead:

Now the agent is a process inside a tmux session on the compute node, and losing your terminal only detaches it. To get back:

Three things to keep straight:

• Tmux session lives on one specific node. You have to be on that node to attach, which is what squeue -u $USER tells you.
• tmux does not extend the allocation. When the walltime in -t runs out, Slurm ends the job and the tmux session with it. Nothing survives the lease.
• Detach on purpose with Ctrl-b d. That leaves everything running. Ctrl-b c opens another window in the same session, which is how you watch a log while the agent works. Those three bindings cover this workflow; any tmux cheat sheet covers the rest.

Long and unattended runs

Interactive jobs are for working. For an evaluation sweep (the same agent over fifty tasks, the kind of thing Assignment 3 onward asks for), you do not want to be present at all. Write a script:

Submit it with sbatch run_eval.sh, which prints a job id and returns immediately. squeue -u $USER tracks it, %j in the output path expands to the job id so concurrent runs do not overwrite each other, and everything the script prints lands in that file. Create the logs/ directory first, in the directory you run sbatch from rather than the one the script cds to, because Slurm resolves -o against the submit directory and opens the file before the script runs. Slurm will not create it for you, and a job whose output file cannot be opened fails instantly for a reason that is hard to see.

Getting your code and data there

• Use git for code. Clone on the login node, commit and push from wherever you worked.
• Use rsync for everything else. From your laptop:
  rsync -avz ./data/ cs2680:~/cs2680/data/ # up rsync -avz cs2680:~/cs2680/results/ ./results/ # down
• OnDemand has a file browser for the one-off case (dragging a CSV in, pulling a plot out), without a terminal at all.
• Keep results out of the node's local disk. Write into your home directory or a shared filesystem. Anything a job leaves in /tmp on a compute node will not be available.

Sharing the HPC

The cluster is small enough that one person’s habits are visible to everyone else, so please be mindful of the rules below. We may cancel a job that is holding far more than its share, particularly in the days before a deadline, so that everyone keeps access to at least one server.

• Ask for what you need. The course pays for the hardware at roughly $3.20 per GPU-hour, so an over-sized request costs money rather than merely lengthening a queue.
• Use the resources responsibly. There is no per-student limit, and you may hold several GPUs when nobody else wants them. However, 50 GPU nodes divided across the enrolled class comes to fewer than two each, so treat that as the share you are borrowing against.
• Bound every job with -t, and prefer a batch job over an interactive one you will not be watching.
• exit when you stop working. An idle allocation is invisible to you and expensive for everyone else, and the queue is longest in the week you most need it.
• Nothing heavy on the login node. It is shared by everyone at once, and the caps are there because someone always tries.

Command reference

Troubleshooting