To fine-tune ZAYA1 with Halo, train Zyphra/ZAYA1-8B on one GPU as a single process, with gradient checkpointing off and bias-update balancing. A shipped config runs it on eight GPUs with expert parallelism (EP8).
Steps
- Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
- Pick the checkpoint.
Zyphra/ZAYA1-8Bis a mixture-of-experts model with 16 routed experts, one selected for each token. It loads from hubmaindirectly, with no revision pin and notrust_remote_code. - Write the config. Create
zaya1-sft.yamlfor supervised fine-tuning of all weights, or use the shippedexamples/sft/zaya/zaya-1-8b-ultrachat.yaml. Among its keys:YAML
model_name_or_path: Zyphra/ZAYA1-8B moe_balancing: bias_update router_balancing_rate: 0.001 attn_implementation: flash_attention_2 max_length: 4096 gradient_checkpointing: false - Launch one process. The recipe starts with one NVIDIA B300 GPU.
Bash
halo launch sft zaya1-sft.yaml -n 1 - Run inference from Transformers. Neither pinned serving engine (vLLM 0.26.0, SGLang 0.5.17) serves a ZAYA1 export.
Common mistakes
- Turning on gradient checkpointing. Gradient checkpointing is unavailable in every mode, and the launcher raises. Keep
gradient_checkpointing: false. - Online reinforcement learning. Online GRPO and async GRPO with environments are refused for this family. Offline GRPO remains available.
With Halo
Halo supports neither context nor tensor parallelism for ZAYA1. The ZAYA1 cookbook has the full config, expert-tensor parallelism on two GPUs, the shipped config for expert parallelism on eight GPUs, and a LoRA adapter.
