Memory and Throughput

Async GRPO is rollout-bound at the step level and logits-bound inside the training forward.

The logits wall

TRL's GRPO computes per-token log-probs from a full [B, T, vocab] logits tensor. At gpt-oss's ~201k vocab, a 24k-token trajectory is ~9 GiB of bf16 logits, roughly doubled by the log-softmax saved for the backward.

Neither EP nor TP shrinks the logits: EP and MoE TP leave lm_head whole, and dense TP gathers its full-vocab output on every rank. Two levers help:

  • Chunked log-probs.
  • per_device_train_batch_size: 1. Both forwards chunk to this many rows, so peak follows the longest single row.

With chunked log-probs off, every GRPO trainer weighs the logits plane against free device memory at construction: rows × (completion width + 1) × vocab × 4 B.

  • Rows: per_device_train_batch_size, or per_device_eval_batch_size when evaluation runs and it is larger.
  • Width: rollout_max_tokens, or max_train_row_tokens under train_on_sampled_tokens: false (the context window when unset). Online and offline GRPO read max_completion_length (use_chunked_grpo_logprobs).
  • Verdict: a plane that cannot fit raises on every rank; one over half the free memory warns.

The estimate is a floor: it covers one forward chunk, a single-row fallback trajectory is wider than a turn, and a model left on the host (accelerate's own FSDP) is weighed against an empty device.

Chunked log-probs

use_chunked_grpo_logprobs: true (default off) removes the wall: log-probs come from the backbone's last_hidden_state through a chunked (sequence × vocab) matmul (halo/kernels/head_logprobs.py, driven by halo/trainers/grpo/logprob_scoring.py), so peak follows the tile size. Every code-contests recipe sets it; online and offline GRPO take the same flag.

  • Precision. Logits and log-probs stay fp32 (the backward takes bf16 operands, fp32 accumulation), within 5e-3 nats of an fp64 reference. A caller's autocast is off inside the sweep, so fp32 activations (QLoRA's) take fp32 head GEMMs in both directions, on the CUDA cores. The full path rounds logits to bf16, and so do both engines by default: on one hidden state that is 0.014 to 0.035 nats mean absolute by family (0.006 to 0.016 on tokens above p 0.1), with signed means of at most 0.005, and up to 0.08 to 0.20 nats at the largest logits (fp32 heads on the engines).
  • Cost. A recompute backward at about the full path's cost. Measured on 1.1.0, whose tiles ran eagerly: Qwen3-8B head at 8k tokens, 51 ms forward+backward on a B300; offline GRPO on 8× B300, Qwen3-8B at 8,192 tokens 18,367 tok/s/GPU against 19,266 on full logits, gpt-oss-20b EP8 at 4,096 tokens 9,042 against 8,980. tests/gpu/profiling/benchmark_head_logprobs.py times the head alone at the Qwen3-8B, gpt-oss-20b and Gemma 4 26B-A4B shapes, a cold compile included.
  • Support. FSDP2, ep1/EP and attention-only TP. A multimodal batch takes the full path. A PEFT adapter on lm_head raises at the first chunked forward.
  • Candidate targets. targets may also be (..., m): column 0 the sampled token, the rest the sampler's candidate ids, each negative id masked (log-prob -inf, no gradient). The m log-probs of a position share its one log Z, the forward gathers them per vocab tile and the backward scatter-adds their gradients with the softmax term weighted by their sum, so no [rows, vocab] plane or [rows, m, vocab] one-hot exists. The form compiles its own forward and backward graph at its first call, which a run with score_centering_top_k makes at construction (Score centering).
  • Compiled tiles. On CUDA everything a tile does after its GEMM runs as one torch.compiled graph per direction: the head transform, the online softmax, the target pick and the entropy forward; the softmax gradient, already in the GEMM dtype, backward. The compile is fullgraph with dynamic shapes, in-process on one compile thread, under inductor's deterministic mode, so ranks that see the same tiles (TP and ETP replicas) and reruns get the same kernels. A compile failure raises. TORCHDYNAMO_DISABLE=1 runs the same tile functions eagerly, at a few 256 MiB tiles more peak memory and more memory traffic, and turns off every other torch.compile in the process with them, FlexAttention's included.
  • Graphs. The first chunked step compiles one graph per direction for the head path (bias, logit scale and softcap each present or not), and one more for the entropy-carrying forward. New row counts, vocabularies, temperatures and head scalars reuse them, as do a tile one row or one column wide (run eagerly), targets sliced out of a wider batch, an expanded upstream gradient, a caller's autocast, and a bias of any dtype, trained or frozen. Each of these compiles one more: a second head path or activation dtype in the process, a caller's inference_mode, toggling torch.use_deterministic_algorithms, and a first tile as tall as it is wide (a vocabulary of at most 4,096). Dynamo allows 8 graphs per tile function; under fullgraph the 9th raises.

The sweep applies the family's head transform (Gemma's softcap, Cohere's logit_scale, Granite's logits_scaling, Inkling's μP division and vocabulary cut), verified against the family's own forward at construction (halo/models/head_transform.py). A mismatch is refused; a new family declares its transform (Adding a Model).

Under FA4, at per_device_train_batch_size: 1, or on an offline GRPO row-position family, rows are trimmed to their real span before the backbone forward, one forward per row, and one head sweep then covers every row's real completion tokens.

One forward per row means one backward recomputes each checkpointed decoder layer once per row. FSDP2's hooks catch only one of those recomputes, so the rest are reduced after the backward, by torch's final callback for a two-layer ZeRO-2 unit and by the trainer for a one-layer group: one extra reduce-scatter per decoder-layer shard group per microbatch, the same size as the one FSDP2 already issues but exposed after the backward, or one per window under fsdp_defer_grad_sync. Until the backward returns, every layer it has recomputed twice stays gathered with an unsharded gradient, ZeRO-3 included (Layers recomputed more than once per backward). With one EP group the experts' divide waits for the optimizer step for the same reason (Gradient synchronization).

Context budget

Keep the trained trajectory under the served context. Reasoning survives only on the last turn:

prompt + last-turn(reasoning + answer) + Σ other-turns(answer + tool output)

One turn's reasoning counts once, not max_turns times (twice under carry_reasoning). prompt + max_turns × rollout_max_tokens is a loose bound that over-restricts max_turns; rollout_max_episode_tokens bounds the generated share directly (Trajectory length).

  • Cap the longest training row with max_train_row_tokens (unset by default; 64000 in the Qwen3.6 curriculum stages).
  • Bound tool output with max_observation_chars (default 16384, in environment_kwargs), which every tool-use environment applies (Common knobs).
  • The usual memory savers are on by default: gradient checkpointing (reentrant on every MoE) and AdamWBF16 at 6 B/param, which a bf16 run gets from the default optim (BF16 optimizer).

Throughput levers

Each step is generate → train → NCCL sync, serial on one server unless it prefetches, so idle trainer GPUs during generation are expected there. In priority order:

  1. Prefetch, the only way to overlap generation with training. It is on by default with two or more servers; a single server needs enable_prefetch: true. At the default prefetch_depth: 1, step time = sync + max(round, update), at one step of IS-corrected staleness, and the servers drain at each round's tail while its longest episodes finish. prefetch_depth: 2 collects the next round beside it to fill that tail, at the cost of one more step of staleness and of engine KV (Prefetch).
  2. Saturate the servers. The optimizer step's row count sets the rollouts in flight; max_concurrent_rollouts only throttles them. Grow the batch with gradient_accumulation_steps (no per-forward memory cost), then raise the cap to match. Waiting: > 0 or timeouts means you overshot.
  3. Speed generation. Server-side levers are on Rollout Servers. Trainer-side, keep rollout_max_tokens and rollout_max_thinking_tokens no larger than the task needs; sync_weights_every_n_steps > 1 amortizes the sync.
  4. Cut max_turns. Turns are sequential, so halving them roughly halves rollout time. If episode/turns is pinned at the cap, raise it; well below, lower it.

nvidia-smi dmon, EfficiencyCallback or async/prefetch_hit_rate shows which phase bounds the step (Prefetch).

Length-ordered micro-batches

Under train_on_sampled_tokens a round holds one row per assistant turn, and at per_device_train_batch_size: 1 each row is its own backbone call: in the loss pass, its checkpoint recompute and the no-grad recompute. Under ZeRO-3 (fsdp_reshard_after_forward: true) or an EP dispatch the ranks forward their rows in lock-step, so a row costs the longest row its peers forward with it. length_ordered_microbatches ranks every rank's rows longest first and deals them round-robin over the round's steps_per_generation micro-batches, in place of TRL's per-rank shuffle: the i-th longest rows of all ranks run together, and each micro-batch takes one row per length stratum. The no-grad forwards run in the same order and hand their scores back in batch order. A rank holding more row tokens in total than its peers still sets the pace.

Only the row order changes. Each row is forwarded on its own, so its log-probs, IS ratio and masks are bitwise the shuffled run's wherever a row's forward does not read its peers' (pure DP, FSDP2, TP; an EP dispatch keeps its own run-to-run tolerance) and the head sweep scores one row at a time (per_device_train_batch_size: 1); under FA4 at per_device_train_batch_size > 1 one head sweep covers a micro-batch's rows together, so a different order matches to kernel rounding. The loss and gradients match the shuffled run's to accumulation order. Unset, the deal is on wherever that holds and off otherwise:

  • accumulation that straddles a generation round, where the order picks which micro-batches an optimizer step takes;
  • loss_type: bnpo, top_entropy_quantile < 1 and the entropy bonus, which read each micro-batch's own tokens;
  • a padded batch (use_chunked_grpo_logprobs: false, or per_device_train_batch_size > 1 without FA4), where every row costs the batch's width whatever its order.

true raises at construction on those; false keeps TRL's shuffle.

On a six-GPU async GRPO test row (1.2.0) the deal takes the update phase from 480.5 to 328.3 s (1.46×), the no-grad pass from 170.7 to 114.0 s (1.50×) and peak memory from 210.25 to 183.35 GiB, the recompute bitwise.

Packed micro-batches

packed_microbatch_tokens replaces one micro-batch per row by token-budget bins. After the round's rewards, advantages and masks exist, each rank plans its real rows into bins of at most that many attended tokens (first-fit-decreasing; a row over the budget keeps its own bin; more bins than steps_per_generation are relaxed to that many balanced ones). The world agrees the largest bin count K and the widest bin, every rank re-partitions into exactly K bins, and the window's last K slots train one bin each. The leading steps_per_generation - K slots are no-ops that run no forward and no collective, so the window still ends on a real backward and FSDP2's window toggles, DDP's no_sync and the optimizer step see the window they see without the knob. The round's tensors stay as built in a round store; TRL's buffer carries only the slot index tensors, and each slot's [rows_per_bin, width] view is gathered from the store just before its loss (filler rows are inert: one attended prompt token, a masked completion, ratio 1).

  • Objective. Row-mean losses (grpo, sapo, dr_grpo, luspo) are rescaled by rows_per_bin × steps_per_generation ÷ round_rows, so a window sums to the per-row run's loss and gradient; steps_per_generation, not gradient_accumulation_steps, is the factor, as each of num_iterations cycles carries the objective once. dapo and cispo divide by a round-global token count and need none. bnpo divides each micro-batch by its own token count and is refused.
  • No-grad passes. The recompute and the KL reference run one scoring call per bin on every rank, an empty bin an inert row that routes naturally and is left out of routing capture and replay. Scores come back in batch order; rows no bin holds (the dummy rows that pad a round across ranks) score 0. A bin forwards only its real rows, not the filler rows the loss forward adds to reach rows_per_bin: a filler row is two attended tokens, so the loss forward's bin holds at most 2 × (rows_per_bin − 1) more tokens than the recompute's, a small share of a bin budget in the thousands, and the grouped GEMM sees correspondingly different per-expert counts. That moves accumulation order only (bf16-level, the same class as the varlen kernel's), not the objective; forwarding the filler too would add its tokens to both no-grad passes for no change in the objective. Its size on a real run is not measured.
  • Packed forward. A bin reaches the backbone as one flattened [1, N] varlen call (one DeepEP capacity scope): each row's positions restart at 0, so it sees the RoPE positions the per-row call gives it, and the family's segment markers (seq_idx, cu_seq_lens) keep the conv, delta-rule and Mamba state per row, as SFT packing emits them. The head sweep then scores each row's real completion tokens. A one-row call carries no markers and runs as the per-row path. Evaluation rounds keep the per-row path, which is why the evaluation batch size must be 1. The result is exact in objective and not bitwise: a varlen kernel and the grouped GEMM's tile choice change accumulation order.
  • Attention and kernels. The script requests the multi-document-rows default under the knob (FA2 for GatedDeltaNet decoders on Blackwell and under full_determinism, the loader's pick otherwise) and keeps the loader's pick for live gpt-oss sinks, which only FA4 carries. A two-document packed forward and backward runs through the scorer at construction on every rank, so the varlen attention and the delta-rule and conv variants load there and not between a dispatch and its combine; the gradients it leaves are dropped and the expert loads it counted are reset.
  • M-RoPE decoders. Qwen3.5/3.6 (text or composite) pack like any other family: each row's positions restart at 0 and the markers keep the GatedDeltaNet state per row, held by the packed-row gate tests/gpu/kernels/test_packed_rl_forward_mrope.py, which scores gradients per parameter group against the per-row run's own bf16 noise (distance from fp32, reordered documents, one backward per document); the per-head scalar group (GatedDeltaNet A_log and dt_bias, token sums of cancelling terms that sit 10% from fp32 per row against 1-2% for every other group) is judged on its packed/per-row norm ratio, inside 0.8-1.25, its direction only logged.
  • Budget. It is checked at construction, on a group spanning NVLink domains, against the proxy-GIN ceiling. A rank whose round tokens exceed steps_per_generation bins of that ceiling raises on every rank.
  • Metrics. TRL's per-micro-step metrics become means over bins, so their logged values shift without any gradient change.

Online GRPO takes the same knob through the same layout (PackedRoundMixin), with the same refusals plus use_sdpg: true (SDPG's OPD term pairs the student's per-row completion logits with a teacher pass sized by the micro-batch; the trainer class declares PACKED_ROWS_REFUSAL, which one blocker reads). TRL, not Halo, runs the old-policy and reference passes there, each as one scoring call over the whole round. While the round builds, those calls plan the rows they are handed (one collective, memoized for the round, so the reference pass and the layout reuse the old pass's bins) and score bin by bin, handing TRL the log-probs in batch order; the layout runs after the advantage hooks and the KL clamp, over the final masks. The loss forward then trains the same bins as slots, so a row's old-policy log-probs and its loss forward come from one bin composition. When neither pass runs (aligned accumulation, no IS correction, beta: 0) the layout plans the round itself. Evaluation rounds keep the padded path.

Rank balance

Online GRPO only. The rollout hands each rank a contiguous slice of the round, so a rank that drew the long completions trains for longer while the rest wait at the step's collectives. balance_rank_tokens deals the round's rows instead: after the advantage hooks and the KL clamp, every rank gathers the world's attended-token counts per row (prompt plus completion) and computes the same deal, heaviest row first onto the lightest rank with a free place, each rank keeping its row count. The rows then move by one all-gather per row tensor, are taken by index through permute_rows, and lose the padding no dealt row needs. The deal runs before the packed layout, so a plan the scoring passes made over the undealt rows is dropped and the layout plans the dealt ones (with packed_microbatch_tokens on, the old-policy and reference passes bin the undealt rows and the loss forward the dealt ones: a bf16 accumulation-order difference, one extra planning collective per round), and before the row spans and any TP broadcast. A round the deal would not make lighter at its heaviest rank comes back untouched. rank_balance/tokens_max_over_mean_before and _after log the heaviest rank's tokens over the mean for each round.

  • Objective. Row-mean and token-global losses (grpo, sapo, dr_grpo, luspo, dapo, cispo) sum the same rows whichever rank holds them, and every rank keeps its row count, so the loss and the gradient equal the unbalanced round's. Other losses, top_entropy_quantile < 1, entropy_coef != 0 and misaligned accumulation are refused, as under the length-ordered deal.
  • Refusals. TP and ETP (the group's ranks share one batch) and a single rank. SDPG deals: its teacher prompts are built before the deal and move with their rows, and a row weighs its prompt, its teacher prompt and the completion twice (once per forward). A round with an image tensor on any rank raises on every rank: only text rows move.
  • Cost. The gather holds one row tensor across the world at a time, so its peak is the world's rows of the widest key, tens of MiB per 1,000 rows of 16k tokens. It adds no step to rounds it leaves in place.
  • Environmental GRPO does not take the knob: its rows are built per rank from that rank's episodes, with trajectory-keyed diagnostics, routing masks and groups kept rank-local, none of which follows a row to another rank.

Weight from the loss forward

On an aligned one-step round every micro-batch's loss forward runs on the weights the round was built on, so the loss forward's own detached log-probs are the log-probs the no-grad pass would return. With loss_forward_importance_weights the pass does not run: the build hands the loss the sampling log-probs, the tokens the ratio corrects and the forced closes it zeroes, and the scoring call of each loss forward builds clamp(exp(logπ − logπ_sampling), max=vllm_importance_sampling_clip_max) from logπ.detach() (a weight with a graph would add A·logπ·∂w/∂θ to the gradient) and clamps the reference log-probs of a beta run against them. The loss and its gradient equal the pass's for every admitted loss type, per-row or packed; they differ only by the numerics between the pass's forward and the loss forward (the entropy-free head graph, no-grad kernel selection, EP's run-to-run dispatch tolerance), a relative change of about the forward's log-prob noise per token.

The knob defaults to false on both trainers. true raises at construction naming the first consumer that needs the round's log-probs before its first backward:

  • balance_token_mass, which needs the round's global importance-weighted token mass;
  • the isr_* stages, isr_engine_reference and skip_update_masked_frac, which pool a trajectory's log-ratio over rows that run in different micro-batches or decide before the first optimizer step;
  • routing_replay: recompute, which pins the update to the pass's routing (rollout fuses: the loss forward replays the engine's masks);
  • num_iterations > 1 and accumulation that straddles a generation round, where later micro-batches run on moved weights and the PPO ratio needs the round-start log-probs.

With early_stop_logratio_gap set, the logged log-ratio reads loss tokens only (below), a different estimator than a threshold set on the pass's reading. Eval rounds keep the pass. With neither the IS correction nor beta there is no pass to remove. sampling/is_weights_from_loss_forward reports the path of each round.

Online GRPO takes the knob too, in place of TRL's old-policy pass (which runs by default under vllm_importance_sampling_correction). The round's build keeps TRL's importance-sampling gate off, so the batch carries the engine's log-probs and no ratio; each loss forward then builds TRL's ratio from logπ.detach() in the configured vllm_importance_sampling_mode, all four of them (token_* per token, sequence_* the exponential of the row's summed log-ratio), with TRL's NaN handling and its truncation or masking at vllm_importance_sampling_clip_min / _max, and clamps the reference log-probs of a beta run. The PPO ratio's denominator is then the forward's own log-probs, so it is exactly 1, as on an aligned round without the correction. The refusals are the shared ones above that apply to it: balance_token_mass, num_iterations > 1, misaligned accumulation, and a run with no IS correction. packed_microbatch_tokens composes, and SDPG's loss reads neither the ratio nor the old log-probs. The gap statistics TRL logs at the build (sampling/sampling_logp_difference/{mean,max}, sampling/importance_sampling_ratio/{min,mean,max}), the token masses and kl_clamp_frac fold at the round's last micro-step and read loss tokens only (rows with a loss token at sequence level).

The loss forward must score exactly once with entropy, and TRL must read the weight and the clamped reference after that call; a forward that bypasses the hook, or a TRL that moves its reads, raises at the micro-batch naming loss_forward_importance_weights: false as the way back. The metrics the pass fed (logratio_mean, is_ratio_*, is_ess_frac, logps/advantage_cov, the token masses, kl_clamp_frac) accumulate on the device over the round's micro-batches and fold in one collective at its last one, before the optimizer step. They read loss tokens only, where the pass read every corrected token, rows the round drops (degenerate groups, invalid or truncated episodes) included. The zero-gradient tracker counts a token as trained unless a forced close zeroed it; a ratio underflowing to 0 (a log-ratio below about −87) is the only difference.

Host work per step

The loss and no-grad loops run once per row at per_device_train_batch_size: 1, so a host read or collective inside them recurs 32 or more times per optimizer step, each draining the launch queue the next op was queued behind. A sync only costs the overlap between the host launching ahead and the device still running, so the gain is bounded by the smaller of the loop's launch time and its device time; on EP lanes DeepEP's own host reads inside each layer already drain the stream. log_phase_times shows where a step goes.

  • Row spans. The dense path trims each row to its attended span, which takes one device-to-host copy of the mask. The no-grad pass reads the spans of its whole batch once, not once per row chunk, and under routing replay the replay layout and the scorer share that read. The round's build also reads them once and attaches them as row_spans, a host table riding TRL's shuffle, split and the length-ordered deal with the other row-aligned keys, so no loss forward copies a mask. The environmental trainer's no-grad capture (step_batch) still reads the spans once per call. Every loss forward recomputes its micro-batch's spans on the device and tallies disagreements without a host read; the stash is always on where rows forward densely and no TP or ETP group exists (no knob), and the window's last micro-step reads the tally (one all-reduce and one host read per optimizer step) and raises on every rank if any row read spans built from another mask, naming the stage to rebuild after (_with_row_spans). Under TP and ETP each micro-batch is re-broadcast from the group leader, so the table is not attached and each forward reads its own spans (logged once at startup).
  • Image consensus. A composite's scoring forwards agree across ranks whether any batch carries images, since mixed full and chunked paths would deadlock FSDP2. The environmental trainer's batches are text-only, so its scoring skips that all-reduce and host read (a modality input reaching it raises on that rank alone, a code-invariant bug trap); online GRPO and SDPG keep one per scoring forward.

TRL's loss forward reduces six metrics across the ranks per micro-step (entropy, three clip ratios and two clip extremes), each ending in a host read; Halo leaves that block as TRL has it.

Long trajectories

A row's training cost grows with its length as attention FLOPs do, and its memory linearly. The update pass on 8 B300s, one row per micro-step (4 per rank), chunked log-probs and gradient checkpointing on:

Row tokensQwen3-8B FSDP2, FA4: s per 1k loss tokensQwen3-8B, FA2Qwen3.5-35B-A3B EP1 ZeRO-3, sdpaPeak GiB, 8B / 35B
16k0.2610.19317.0 / 32.5
32k0.3680.20225.6 / 38.9
64k0.2700.5810.26042.8 / 52.6
128k0.3971.0100.39977.5 / 80.5

What keeps long rows cheap:

  • A varlen flash backend. FA4 runs the 8B 2.2× faster than FA2 at 64k and 2.5× at 128k, at the same peak; FA2's attention falls below GEMM efficiency as its share of the FLOPs grows (27% at 16k, 75% at 128k).
  • Weight from the loss forward. The separate no-grad pass it skips takes 23 s per 4 rows of 128k on the 8B with FA4 (55 s with FA2) and 22 s on the 35B.
  • Packed micro-batches cost nothing at one row per bin (64k measured), so a round of mixed lengths keeps them on for its short rows.
  • Routing replay. routing_replay: recompute holds the captured routing on the host and pads only the rows a forward runs: on the 35B at 24 rows of 64k it adds 0.16 GiB to the update's peak and 0.94 GiB to the capture pass.
  • EP. Within one NVLink domain a row has no dispatch ceiling; across domains the dispatch token ceiling applies.

A row over max_train_row_tokens leaves the batch.

Sizing a run

For a new model, environment or GPU, size in order:

  • Rollouts in flight per server = rollout ranks × min(max_concurrent_rollouts, per_device_train_batch_size × steps_per_generation) ÷ num_servers per round, peaking at a round's start, with up to prefetch_depth rounds in flight at once; an eval round opens with rollout ranks × eval_rollout_batch_size ÷ num_servers (eval rounds). Rollout ranks are every rank, or under TP/ETP each group's leader alone, as its followers take the leader's rollouts (Ray — Pool sizing).
  • Decode speed at that concurrency. Engine step latency ≈ a + b × n (one CPU thread). Measure two concurrencies against a live server with the run's own prompts (Throughput).
  • Timeouts from the per-turn cap, the turn's total (Reasoning cap): rollout_max_tokens, or the engine's reasoning cap plus rollout_max_answer_tokens where that is smaller.
    • request_timeout ≥ 2 × cap ÷ speed(n_peak) (default 120).
    • episode_timeout ≥ min(max_turns × cap, rollout_max_episode_tokens) ÷ speed(n_peak) + tool time (default 1200); over the NCCL watchdog it raises at startup.
  • KV capacity. bf16 KV per token = 2 × full-attention layers × kv_heads × head_dim × 2 B, 20 KB on Qwen3.6-35B-A3B. Check n_peak × mean context × that against the pool the server reports at startup.
  • Step time. The round is the batch's slowest episode. The update is per-rank episodes × per-episode train time, read off a short run, and hides inside the round while shorter.
  • Serving against training. A trainer GPU absorbs about 3600 ÷ per-episode train time episodes per hour. Keep the two within 2×: roughly one serving GPU per trainer GPU, each running one TP=1 engine while the model and its KV fit.

Worked shape, the 4-rank code-contests recipe (Qwen3.6-35B-A3B, 2 servers, gradient_accumulation_steps: 24, num_generations: 8):

QuantityValue
Episodes per step96 (12 prompts), 48 per server; an eval round (eval_rollout_batch_size: 50) opens with 100 per server
Per-turn capThe 30,000-token rollout_max_tokens. The engine closes a normal turn's reasoning at 21,808 at every level, leaving the 8,192-token answer room
Timeoutsrequest_timeout: 1200, episode_timeout: 2700
KV48 × 13k × 20 KB ≈ 12 GB at a 13k mean context; 48 × 30k × 20 KB ≈ 29 GB once every request holds a whole turn; about twice both in an eval round

Source on GitHubUpdated 18 November 2026