Fine-tuning embedding models

Embedding fine-tuning trains a model that turns text into vectors so that related texts land close together and unrelated ones far apart. It improves retrieval and similarity search on your own data.

How it works

An embedding model maps a text to one vector, and two texts are compared by the similarity of their vectors, usually cosine. Training is contrastive: it pulls an anchor toward its positive and pushes it away from negatives. The data comes in three shapes:

DataTrains
Pairs: a query and a matching passageRetrieval. Other passages in the batch act as negatives.
Triplets: a pair plus a hard negativeRetrieval, against passages that look relevant and are not.
Scored pairs: two texts and a scoreGraded similarity.

With in-batch negatives, a larger batch size gives each query more negatives and a harder task.

When to use it

Tune the embedder when search misses your domain's vocabulary: product names, legal or medical terms, code. It is a direct fix for retrieval-augmented generation (RAG) that fetches the wrong passages. See fine-tuning vs RAG.

In Halo

Halo trains embedders with sentence-transformers losses, set by loss_type. The default, mnrl, uses in-batch negatives. Neither gradient accumulation nor extra GPUs enlarge that pool, so raise per_device_train_batch_size. triplet is for mined hard negatives, and cosent for graded scores.

Bash

halo launch embedding examples/embedding/qwen3/embedding-qwen3-4b-nq.yaml

The output loads with SentenceTransformer(path). See the embedding guide.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL