High-level design · Dynamo
Global Router
The router we already have, run one level up: each pool acts as one super-worker.
Click a pool to pick where it goes.■ global level ■ pool level ■ KV & load
Next → the router picked a pool. What does it know about each pool?
Pool signals
What the global router knows about a pool
PoolRelay publishes which blocks the pool has and its load. Which worker has them stays private.
Next → turning those signals into a choice of pool
Selection
Picking a pool: filter → rank → admit
Filter and rank run in the global frontend’s router. Admission is the pool’s final say.
Next → the router only picks among known pools. How does a pool become known?
Discovery
How a new pool joins
PoolRelay registers the whole pool, frontends included. Every global frontend ends up with the same record. The mode only changes the path.
Next → the pool is known. Which connections get opened, and by whom?
Replicas & connections
PoolRelay dials out. Global frontends dial in.
Every PoolRelay replica opens telemetry to every global frontend. Global frontends open serving links to the pool’s frontends.
Global frontend 1 · pool directory
Next → how a pool stays registered, and what failures can remove it
Leases & failure handling
Losing signals never deletes a pool
Each frontend lease, renewed by its own PoolRelay replica, expires on its own. Only expiry, removal or drain changes membership.
Global frontend’s view of Pool A
Next → membership survives failures. What about the request in flight?
Migration
A pool fails. The answer keeps going.
The global frontend replays prompt + tokens so far to another eligible pool.
Last request the global frontend sent
Next → removing a pool on purpose
Drain
Drain: stop new work, finish accepted work
The pool rejects locally. Global frontends catch up when they catch up.
Deadline
Next → how much of this already exists
Implementation boundary & scope
Mostly reuse. The new work is the seams.
Everything in this deck is the first release. The dashed row comes later.