FlashAdamW (Quantized Optimizer States)

FlashAdamW is a drop-in replacement for AdamW that cuts per-parameter optimizer memory ~58% via 8-bit quantized Adam moments and a 24-bit master weight, with quantize/dequantize fused into the update by Triton kernels. Convergence matches AdamW.

Per-param state: AdamW 12 B (4B exp_avg + 4B exp_avg_sq + 4B master, all fp32) vs FlashAdamW ~5 B (1B int8 exp_avg + 1B int8 exp_avg_sq + 3B master = bf16 param as the high bits + an 8-bit low-bit correction). At 20B params: 240 GB vs ~100 GB optimizer state.

Usage

YAML

optim: flash_adamw
learning_rate: 1e-4
weight_decay: 0.01

Bash

torchrun --nproc_per_node=8 \
    scripts/training/sft.py examples/sft/qwen3/qwen3-4b-ultrachat.yaml \
    --optim=flash_adamw --learning_rate=1e-4 --weight_decay=0.01

Both images install flashoptim at its locked version with the rest of uv.lock (the image's export carries the flash-optimizers extra), so optim: flash_adamw works out of the box. On a bare host, install the extra (resolves from PyPI):

Bash

uv pip install -e ".[flash-optimizers]"   # or: pip install flashoptim

Without the package, optim: flash_adamw raises an import error stating what to install.

The YAML/trainer path (flash_adamw_optimizer_from_args in halo/optimizers/flash_adamw.py) wires learning_rate, weight_decay, betas (from adam_beta1/adam_beta2), and eps (from adam_epsilon). Only master_weight_bits is not read from TrainingArguments, so over YAML it stays at its default. To override it, construct directly: build_flash_adamw_optimizer(model, lr=..., weight_decay=..., master_weight_bits=24) in halo/optimizers/flash_adamw.py. The wrapper splits parameters into weight-decay groups (decay_parameters get weight_decay, the rest 0.0).

build_flash_adamw_optimizer defaults: lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0.01, master_weight_bits=24 (24 = bf16 + 8-bit correction, 32 = full fp32, None = no master weights).

Benchmark

The value is memory, and it scales with model size — tens of GB at 70B+ params, where it is the lever.

Optimizer micro-benchmark on B300, Halo 1.1.0 or earlier (tests/gpu/optimizers/bench_muon.py --hidden 4096 --layers 8, synthetic FFN): FlashAdamW peak memory is −8.6% vs torch.optim.AdamW(fused=True) over the same bf16 parameters (bf16 states), at a ~5 ms optimizer-step overhead from state quant/dequant. For the AdamW / AdamWBF16 / Muon comparison at the same shape, see Muon.

That overhead amortizes against fwd+bwd on most models, but it is not free — on an attention-bound long-context step the per-step cost can outweigh the memory saving (GLM-4.7-Flash EP8 32k, Halo 1.1.0 or earlier: ~50 s/step vs ~34 s/step on the default optim, which bf16: true resolves to AdamWBF16).

A real-model benchmark lives in tests/gpu/optimizers/bench_muon_qwen3_5.py (Qwen3.5-2B). Correctness is covered by tests/gpu/optimizers/test_flash_adamw.py — creation, loss descent, decay filtering, memory savings, and a bit-exact state-dict round-trip.

When to use which optimizer

OptimizerMemoryConvergenceBest for
adamw_torch_fused under bf16: false12B/paramBaselinefp32 runs; refused at ep_group_size > 1 or at ep1 with fsdp_shard_ep1_experts: false, unless fp32_non_ep_params is set
AdamWBF16 (auto: the default adamw_torch_fused under bf16: true)6B/paramSame as FP32Standard bf16 training
flash_adamw~5B/paramSame as AdamWMaximum memory savings
muon~4–6B/param—Orthogonalized updates on matrix params — see Muon

Compatibility

optim: flash_adamw routes through create_optimizer (build_optimizer, halo/trainers/construction/optimizer_selection.py) for every distributed trainer, the same path as Muon and AdamWBF16, and composes with gradient checkpointing and FSDP2. flashoptim steps per parameter and unwraps DTensors to their local shard before the kernel runs, so a weight-decay group mixing FSDP2/TP DTensors with the plain rank-local tensors EP and fp32_non_ep_params produce is handled, not rejected — examples/sft/laguna/laguna-s-2.1-ultrachat-ep.yaml ships this combination at expert_parallel_size: 8.

Pipeline parallelism refuses it at trainer construction (reject_pp_named_optimizer, halo/trainers/parallelism/pipeline_rules.py): no PP run validates it. The other guard is the shared reject_conflicting_optimizer_choice: bf16_optimizer: true together with a named optim raises rather than silently discarding one of them. Correctness coverage: test_flash_adamw.py on one GPU, and test_flash_adamw_parallel.py on two through the trainer (EP2 with a mixed DTensor/plain group, FSDP2, and the uneven-shard save refusal).

The non-finite step guard reads the gradient norm back before every FlashAdamW step and skips a non-finite one on the host: flashoptim's fused kernel takes no device skip flag, so FlashAdamW keeps the one host sync per step that AdamWBF16 and Muon avoid.

The wrapper registers a step post-hook that advances each stepped param's version counter — flashoptim's fused Triton update stores through raw pointers ATen never sees, and the low-precision weight-quant cache keys on that counter (see Low-Precision MoE).

Checkpoint limit: unevenly sharded params. flashoptim's state_dict refuses any FSDP2 DTensor param whose sharded dim does not divide the mesh, and the trainer's all-or-nothing shard save then raises at the first checkpoint. Training itself is unaffected. _warn_if_uneven_shards (halo/optimizers/flash_adamw.py) says so at optimizer build; set save_only_model: true for weights-only checkpoints, or pick dims divisible by the shard world or another optim to keep optimizer state.

References

  • flashoptim — Databricks' implementation; pyproject.toml allows >=0.1.3,<0.2.0, uv.lock and so the images pin 0.1.4
  • 8-Bit Optimizer Paper — dynamic quantization for Adam states

Source on GitHubUpdated 18 November 2026