# How to fine-tune ZAYA1

To fine-tune ZAYA1 with Halo, train Zyphra/ZAYA1-8B on one GPU as a single process, with gradient checkpointing off and bias-update balancing. A shipped config runs it on eight GPUs with expert parallelism (EP8).

## Steps

1. **Start the training container.** Set up the host as [installation](/halo/docs/installation) describes, then start the [cookbook container](/halo/docs/cookbooks#start-the-training-container). Run the commands below inside it.
2. **Pick the checkpoint.** `Zyphra/ZAYA1-8B` is a [mixture-of-experts](/glossary/mixture-of-experts) model with 16 routed experts, one selected for each token. It loads from hub `main` directly, with no revision pin and no `trust_remote_code`.
3. **Write the config.** Create `zaya1-sft.yaml` for supervised fine-tuning of all weights, or use the shipped `examples/sft/zaya/zaya-1-8b-ultrachat.yaml`. Among its keys:
   ```yaml
   model_name_or_path: Zyphra/ZAYA1-8B
   moe_balancing: bias_update
   router_balancing_rate: 0.001
   attn_implementation: flash_attention_2
   max_length: 4096
   gradient_checkpointing: false
   ```
4. **Launch one process.** The recipe starts with one NVIDIA B300 GPU.
   ```bash
   halo launch sft zaya1-sft.yaml -n 1
   ```
5. **Run inference from Transformers.** Neither pinned serving engine (vLLM 0.26.0, SGLang 0.5.17) serves a ZAYA1 export.

## Common mistakes

- **Turning on gradient checkpointing.** [Gradient checkpointing](/glossary/gradient-checkpointing) is unavailable in every mode, and the launcher raises. Keep `gradient_checkpointing: false`.
- **Online reinforcement learning.** Online GRPO and async GRPO with environments are refused for this family. [Offline GRPO](/glossary/offline-grpo) remains available.

## With Halo

Halo supports neither context nor tensor parallelism for ZAYA1. The [ZAYA1 cookbook](/halo/docs/cookbooks/halo-zaya1-cookbook) has the full config, expert-tensor parallelism on two GPUs, the shipped config for [expert parallelism](/glossary/expert-parallelism) on eight GPUs, and a [LoRA](/glossary/lora) adapter.

