GTC EUROPE - BERLIN - OCTOBER 2026
Post-training open models into computer-use agents
Online RL with prime-rl, and what NVIDIA Dynamo changes in the inference loop
Emrick Sinitambirivoutin, Core Team Research Engineer, H Company
hcompany.ai
TRAINING PIPELINE
FROM AN OPEN MODEL TO AN AGENT
Post training @ H Company
How we specialize models for computer use
REF hcompany.ai/newsroom/holo4 (2026-09-28)
Agentic Task Factory
coding agents build envs + verifiers · ≈10,000 tasks
Open base model
Qwen3.8 27B
Nemotron 3 Nano Omni
SFT
Trained on 127B tokens, from our Agentic task factory
(GUI, API, MCP)
Online RL
Trained on live environments with verifier rewards, GRPO and LoRA adapters
Merge
experts folded back
at equal weight
no further training
Holo4 · Holotron4
desktop · web · mobile
MCP · API · terminal
SFT MIXTURE, SHARE OF 127B TOKENS
45%
Desktop trajectories
14%
Web
12%
MCP + API
Mobile 3%
26%
Grounding · reasoning · tool use
01 / 14
INFERENCE FLEET
Frontend / router
Engine workers, ranks 0…N
routes requests
PRIME-RL
HOW ONLINE RL IS WIRED
prime-rl high level architecture
3 components that run independently
REF prime-rl (github.com/PrimeIntellect-ai/prime-rl)
Orchestrator
Trainer
1 discovery
2 rollout requests
3 admin plane
pause · load adapter · resume
4 weight transfer
5 training batch
02 / 14
INFRASTRUCTURE TODAY
REF SkyPilot · github.com/skypilot-org/skypilot · docs.skypilot.co
1 · sky launch
API server queues
the request
2 · Provision
N pods on Kubernetes (EKS)
each 8×H100 · 128 CPU · 1.9 TB
3 · Bootstrap
Ray cluster + skylet per pod
S3 / EFS mounts · code sync
4 · Launch
one Ray task per pod runs
setup, then run (NODE_RANK)
SKYPILOT CLUSTER · ONE PER RUN
rank 0
Inference
vLLM + router
+ orchestrator
Ray · skylet
rank 1
Trainer
Ray · skylet
rank 2
Trainer
Ray · skylet
rank 3
Trainer
Ray · skylet
rank 4
Trainer
Ray · skylet
FIXED UNTIL THE RUN ENDS
The inference fleet is fixed, which means:
• We cannot adjust inference replicas to the load
• A failing inference replica fails the job
03 / 14
HOW A RUN IS LAUNCHED
Prime-rl @ H Company
How we use Skypilot to orchestrate prime-rl in our infrastructure
INFRASTRUCTURE WITH DYNAMO
REF NVIDIA Dynamo · github.com/ai-dynamo/dynamo · SkyPilot · github.com/skypilot-org/skypilot
SKYPILOT CLUSTER · ONE PER RUN
rank 0
Trainer + orchestrator
Ray · skylet
rank 1
Trainer
Ray · skylet
rank 2
Trainer
Ray · skylet
rank 3
Trainer
Ray · skylet
No inference pods: rollouts go to the shared fleet.
KUBERNETES · SHARED INFERENCE CLUSTER
Dynamo frontend
OpenAI API · Rust chat processor · KV-aware router
/v1/rl/workers discovery
Worker 0
Worker 1
+ worker
joins or leaves
mid-run, loads the
current adapter
EFS
shared adapters
etcd
worker discovery
ModelExpress
fast boot over RDMA
rollouts (chat, MITO)
discovery
load adapter, each
adapter files
04 / 14
WHAT CHANGES
Prime-rl @ H Company with Dynamo
How we use our shared inference infrastructure for RL
OPTIMIZATIONS
THE STARTING POINT
Where the time went
Inconsistent prefixes slowed training and inference
REF prime-rl (github.com/PrimeIntellect-ai/prime-rl) · means over training steps 3–50 of the first run (3-screenshot window, 2 vLLM servers)
05 / 14
ONE TRAINING STEP: 285 S
forward + backward 217 s
waiting for rollouts 57 s
The trainer computes 80% of the step. Two things made it slow:
01
Every turn is its own training sample
About 24 turns per rollout, each trained alone: the trainer re-reads the whole history every turn.
TOKENS THE TRAINER READS PER ROLLOUT
prompt, re-read: 98%
completion, trained: 2%
02
The prompt changes at every turn
The 3-screenshot window drops an image each turn: turns can't share a sample, and the cache is invalidated.
PREFIX-CACHE HIT ON THE INFERENCE SERVERS
0.73
Next: an append-only prompt (folded history), then one sample for consecutive turns (turn merging).
MULTIMODAL RL
FIXING THE TRAINER · 1 OF 2
Folded screenshot history
Invalidate the prefix cache once every 5 turns, not every turn
REF Folded history as in Qwen-CUA (keeps 20 screenshots, folds 10); ours keeps 10, folds 5 · vLLM automatic prefix caching
06 / 14
WHAT THE PROMPT OF TURN 7 REUSES FROM TURN 6
Rolling window
last 3 screenshots
1
2
3
4
5
6
7
turn 4 lost its screenshot → recompute KV cache from there (4 turns)
Folded history
up to 10, then drop 5
1
2
3
4
5
6
7
nothing changed: only the new turn is computed
CACHE INVALIDATIONS OVER THE FIRST 20 TURNS
Rolling window
17 invalidations
Folded history
2 invalidations
1
4
11
16
20
prompt extends the last one
prompt changed: recompute
Longer prompts, but the cache survives 4 turns out of 5.
W&B · PREFIX-CACHE HIT PER STEP
PREFIX-CACHE HIT
0.73 → 0.83
MULTIMODAL RL
FIXING THE TRAINER · 2 OF 2
Turn merging
A 22-turn rollout trains as 4 samples instead of 22
REF prime-rl (github.com/PrimeIntellect-ai/prime-rl) · token bars from one recorded rollout, first 10 turns
07 / 14
Each turn as its own sample
turn 1
turn 2
turn 3
turn 4
turn 5
turn 6
turn 7
turn 8
turn 9
turn 10
120,100 tokens through the trainer
prompt: read, no loss
The same turns as one merged sample
16,590 tokens, 7.2× fewer
completion: trained
A turn joins the previous sample only if its prompt extends it exactly: we fixed the chat template's re-rendering of past reasoning, in our stack and in Dynamo's frontend.
W&B · TRAINING SAMPLES PER ROLLOUT
TOKENS PER TRAINING STEP
22.6M → 4.7M
FORWARD + BACKWARD
259 → 89 s
Now inference-bound: the trainer waits 144 s per step for rollouts.
NVIDIA DYNAMO
FIXING INFERENCE · ROUTING
Routing multi-turn rollouts
Pin each rollout to one worker: prefix-cache hit 0.62 → 0.81
REF NVIDIA Dynamo KV router (github.com/ai-dynamo/dynamo), DYN_ROUTER_SESSION_AFFINITY_MODE=hard · figure: illustration
08 / 14
session-aware:
X-Session-ID → A
KV-aware:
lighter load wins → B
Turn 9 of rollout R
Dynamo router
Worker A
KV CACHE
R, turns 1–8
other rollouts
RUNNING REQUESTS
24
prefix reused: only turn 9 is prefilled
Worker B
KV CACHE
other rollouts
free
RUNNING REQUESTS
12
prefix recomputed: turns 1–8 prefilled again
STEPS 3–50
CACHE HIT
STEP
GPU-hours / 100
prime-rl vLLM, 4 servers
0.83
192 s
213
Dynamo, KV-aware
0.62
192 s
214
Dynamo, session-aware
0.81
168 s
188 (−12%)
W&B · PREFIX-CACHE HIT PER STEP
AUTOSCALING
FIXING INFERENCE · AN ELASTIC FLEET
Scaling on the trainer's wait
Load-based auto scaling can't see the trainer waiting
REF NVIDIA Dynamo (github.com/ai-dynamo/dynamo) · Kubernetes Horizontal Pod Autoscaler · run with the load-based autoscaler, steps 3–100
09 / 14
The problem with scaling on load
What the autoscaler sees
20–25 requests per worker, under its target of 30: it never scales up.
REQUESTS PER WORKER
target 30
What the trainer sees
It waits 51 s of every 153 s step for its slowest rollouts.
ONE STEP: 153 S
51 s
waiting for rollouts
Solving it with a custom controller
INPUTS, EVERY 5 MIN
Trainer
step time, wait
Workers
load, latency
Rollouts
time in model calls
OUTPUT
Autoscaler
min = max = N
Cost-aware controller
1
Fit latency vs. requests per worker
2
Predict the step time for each fleet size
3
Price each size in GPU-seconds per step
4
Pick the fastest within 3% of the cheapest
Dynamo fleet
N vLLM workers
new metrics
AUTOSCALING
FIXING INFERENCE · AN ELASTIC FLEET
The controller follows the trainer
The HPA holds its fleet; the controller scales with the wait
REF NVIDIA Dynamo (github.com/ai-dynamo/dynamo) · Kubernetes Horizontal Pod Autoscaler · live runs, training steps 3–100
10 / 14
HPA on load
target 30 requests per worker
4
8
workers
trainer wait
WORKERS, MEAN
4.1
Controller
trainer-aware, cost-aware, max 10 workers
4
8
workers
trainer wait
WORKERS, MEAN
7.7
step 3
25
50
75
100
NVIDIA DYNAMO
FIXING INFERENCE · FAST BOOTS
ModelExpress
A new worker serves in 2.6 minutes instead of 11.1 minutes
REF NVIDIA ModelExpress (github.com/ai-dynamo/modelexpress) · NIXL (github.com/ai-dynamo/nixl) · vLLM (github.com/vllm-project/vllm)
11 / 14
image pull
process start, registration
weights
engine init: compile, profile, graphs
ModelExpress + warm cache
67
85
2.6 min
Weights from S3, no cache
59
190
172
7.0 min
… and image not on the node
242
59
190
172
11.1 min
DECISION → SERVING
2–3 min vs 6–8
FLEET REACHES 8 WORKERS
step 28 vs 34
STEP TIME, SAME GPU-HOURS
119 s vs 125
RESULTS
SAME CODE, 100 STEPS EACH
Results
Faster training (×1.24) at lower cost (5% less)
REF prime-rl · NVIDIA Dynamo · ModelExpress · trainer on 4 nodes × 8 H100 in every setup · GPU-hours include the trainer's idle time
12 / 14
SETUP
INFERENCE GPUS
STEP TIME (speedups)
TRAINER WAIT
100 STEPS
GPU-Hours / 100 STEPS (savings)
prime-rl
4 vLLM servers (TP2), fixed
8
147 s
51 s
4.1 h
164
Dynamo + controller
trainer-aware controller sizes the fleet
13.1
125 s ×1.18
20 s
3.4 h
157 −4%
Dynamo + controller + ModelExpress
fast worker boots
14.8
119 s ×1.24
14 s
3.3 h
155 −5%
RESULTS
THE SAME RUNS IN W&B
Training step vs wall-clock
Step 100 at 3.3h vs 4.1h for prime-rl
REF Weights & Biases · minutes since each trainer started; the results table times steps 2–100
13 / 14
W&B · TRAINING STEP VS MINUTES SINCE START
TAKEAWAYS
WHAT TO REMEMBER
REF hcompany.ai · github.com/PrimeIntellect-ai/prime-rl · github.com/ai-dynamo/dynamo
Thank you
14 / 14
A production customer's RL setup, redrawn. Each number is one of the next three demos.
One KV-aware router
across clusters and regions
Dead replica → reroute session
trajectory survives
>99% fewer bytes over the WAN
3–4 s inference pause per update
All four datacenters act as one pool. No fixed share of the batch per site.
A KV-cache hit makes the turn cheap. Slow GPUs cost more, so long rollouts leave them.
No datacenter sits idle waiting on the slowest site, so training starts sooner.
If the GPU dies, the session’s history survives in the proxy.
Sent to the least-loaded healthy worker. Only the current turn is redone.
Loads a checkpoint plus small Δ updates, not the full 2 TB model.
Mostly zeros → >99% smaller
One upload per step
Peers copy over NIXL, not the WAN
Swap weights, clear KV cache