To red team a large language model (LLM), define what must not happen, build a set of attacks for each risk, run them against the whole system, score the outcomes, fix what failed and rerun on every deploy.
Steps
- Define what must not happen. Write each failure as a concrete outcome: the system prompt is revealed, another user's data appears, the agent sends an email nobody asked for.
- Build an attack set per risk. Collect prompts that try to cause each failure: jailbreaks, direct and indirect prompt injection, requests for data, long multi-turn conversations. Add variants in other languages and encodings.
- Run them on the whole system. Test the app as deployed, with its system prompt, retrieval, tools and guardrails. A model that refuses a request on its own can still fail when the same request arrives inside a retrieved document.
- Score the outcomes. Record whether the failure happened, not whether the reply sounded like a refusal. Use rules or a judge model for volume, and check a sample by hand. Report the attack success rate per risk, with the inputs that worked.
- Fix. Add or tighten a guardrail policy, change the prompt, or remove a permission. Then rerun the attacks that got through.
- Rerun on every deploy. Keep the attack set as a regression suite. A new model version, prompt or tool can reopen a hole that was closed. See continuous red teaming.
With White Circle
Red Team simulates real-world attacks on models, retrieval-augmented generation (RAG) pipelines, tool-using setups and agents, in one-off campaigns or continuously with regressions tracked over time. Findings can be patched through Control in one click.
