How to red team an LLM

To red team a large language model (LLM), define what must not happen, build a set of attacks for each risk, run them against the whole system, score the outcomes, fix what failed and rerun on every deploy.

Steps

  1. Define what must not happen. Write each failure as a concrete outcome: the system prompt is revealed, another user's data appears, the agent sends an email nobody asked for.
  2. Build an attack set per risk. Collect prompts that try to cause each failure: jailbreaks, direct and indirect prompt injection, requests for data, long multi-turn conversations. Add variants in other languages and encodings.
  3. Run them on the whole system. Test the app as deployed, with its system prompt, retrieval, tools and guardrails. A model that refuses a request on its own can still fail when the same request arrives inside a retrieved document.
  4. Score the outcomes. Record whether the failure happened, not whether the reply sounded like a refusal. Use rules or a judge model for volume, and check a sample by hand. Report the attack success rate per risk, with the inputs that worked.
  5. Fix. Add or tighten a guardrail policy, change the prompt, or remove a permission. Then rerun the attacks that got through.
  6. Rerun on every deploy. Keep the attack set as a regression suite. A new model version, prompt or tool can reopen a hole that was closed. See continuous red teaming.

With White Circle

Red Team simulates real-world attacks on models, retrieval-augmented generation (RAG) pipelines, tool-using setups and agents, in one-off campaigns or continuously with regressions tracked over time. Findings can be patched through Control in one click.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL