Fine-tuning vision-language models

Vision-language fine-tuning trains a model that reads images and text on examples that pair images with the answer you want. It is supervised fine-tuning with image tokens in the sequence.

How it works

Most vision-language models (VLMs) pair a vision encoder with a language model. The encoder turns an image into a sequence of embeddings, and the language model reads them in line with the text tokens.

Training is supervised fine-tuning. Each example is a conversation with one or more images, and the loss is next-token cross-entropy on the answer. Typical tasks are document question answering, chart and screenshot reading, and captioning in a narrow domain.

What to watch

  • Length. One image can take hundreds or thousands of tokens, so rows are long. Budget context length and memory for them.
  • Truncation. A row cannot be cut in the middle of an image, because the image tokens must match the pixels. Filter out rows that are too long.

In Halo

Nothing to switch on. halo launch sft takes the VLM path when a multimodal checkpoint meets a dataset with images. Images ride inside message content, or in a column named by images_field, as in examples/sft/qwen3_5/qwen3.5-9b-vl-docvqa.yaml:

YAML

dataset:
- HuggingFaceM4/Docmatix:zero-shot-exp
conversation_field: texts
images_field: images
max_length: 28672

An image run refuses packing and context parallelism, and the collator raises rather than truncating a batch over max_length. Vision data also works for preference tuning, reward modeling and both distillation trainers. See the supervised fine-tuning guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL