8 · How Snapshot works on Kubernetes

MX alone: 180 s
Snapshot alone: 31 s
Together: 14 s

Key message: Checkpoint on one node, restore on any node. A per-node agent does all the work from the host.

Talk track (follow the numbers) 1. ① Engine init: the worker pod starts normally: loads weights, compiles kernels, captures CUDA graphs. 2. ② ready-for-checkpoint: before joining the Dynamo runtime, the worker writes a signal file (used as its readiness probe) and waits in a polling loop. 3. ③ snapshot-agent (a privileged DaemonSet, installed with Helm, on every node) runs: - cuda-checkpoint, which copies GPU state into host memory - CRIU, which dumps the process tree to files, plus the container's overlay filesystem - It then writes the image to shared storage (PVC/NFS). 4. ④ read: on Node B, the agent reads the image and starts a lightweight placeholder pod. 5. ⑤ restore: the agent restores the overlay filesystem, the CRIU process and the CUDA state into that pod. 6. ⑥ restore-complete → register: the worker resumes at the exact instruction it paused on, sees the signal, connects to the control plane and registers. The router starts sending traffic.

Why it's designed this way - Clean image: checkpointing before joining the cluster means no TCP or RDMA connections are captured, since those can't survive a move to a new pod IP. - Portable: the image lives in shared storage, not on the node. - Non-invasive: no runc changes, and no reliance on Kubernetes checkpoint feature gates.

Likely questions - "Can it restore on any GPU?" Only on compatible nodes (same GPU type and driver stack). - "Driver requirements?" For the best performance on large models the Snapshot team recommends a newer user-mode driver (r615). Check with them for your cluster.