How to fine-tune ZAYA1

To fine-tune ZAYA1 with Halo, train Zyphra/ZAYA1-8B on one GPU as a single process, with gradient checkpointing off and bias-update balancing. A shipped config runs it on eight GPUs with expert parallelism (EP8).

Steps

  1. Start the training container. Set up the host as installation describes, then start the cookbook container. Run the commands below inside it.
  2. Pick the checkpoint. Zyphra/ZAYA1-8B is a mixture-of-experts model with 16 routed experts, one selected for each token. It loads from hub main directly, with no revision pin and no trust_remote_code.
  3. Write the config. Create zaya1-sft.yaml for supervised fine-tuning of all weights, or use the shipped examples/sft/zaya/zaya-1-8b-ultrachat.yaml. Among its keys:

    YAML

    model_name_or_path: Zyphra/ZAYA1-8B
    moe_balancing: bias_update
    router_balancing_rate: 0.001
    attn_implementation: flash_attention_2
    max_length: 4096
    gradient_checkpointing: false
  4. Launch one process. The recipe starts with one NVIDIA B300 GPU.

    Bash

    halo launch sft zaya1-sft.yaml -n 1
  5. Run inference from Transformers. Neither pinned serving engine (vLLM 0.26.0, SGLang 0.5.17) serves a ZAYA1 export.

Common mistakes

  • Turning on gradient checkpointing. Gradient checkpointing is unavailable in every mode, and the launcher raises. Keep gradient_checkpointing: false.
  • Online reinforcement learning. Online GRPO and async GRPO with environments are refused for this family. Offline GRPO remains available.

With Halo

Halo supports neither context nor tensor parallelism for ZAYA1. The ZAYA1 cookbook has the full config, expert-tensor parallelism on two GPUs, the shipped config for expert parallelism on eight GPUs, and a LoRA adapter.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL