Philosophy

AI systems can now act on their own, and increasingly do. They move money, write and ship code, answer the inbox, browse the web and act on what they find. They are getting cheaper and more capable, so there are more of them, working in groups, instructing one another, over longer horizons.

Every one of those actions rests on a judgment: what did the person want, what does the certain context allow and consider appropriate, what should never happen whoever is asking? The system makes that judgment on the go, for each prompt and constructed response, at speeds humans are not keeping up with.

A mathematician saw this problem clearly as early as 1960, writing about machines that played checkers:

If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.

— Norbert Wiener, Some Moral and Technical Consequences of Automation, 1960

There are two ways out of Wiener's problem. Be quite sure of the purpose before you start, which is what most alignment research is for, and which nobody has finished. Or get the data to intervene before the action is complete. We work on both ways out.

White Circle builds independent control over AI so that we can prevent catastrophic outcomes.

We believe the following things.

1. A model does not have to be hostile to be dangerous

At first, a coding assistant suggests a line and the developer accepts it. Then, a coding agent opens a pull request and a human merges it. In the end, a coding agent plans the change, writes it, tests it, merges it and deploys it, and the human reads about it in the morning. The economic pressures are such that both individuals and organizations will hand more decisions to these systems.

One obvious worry about this is misalignment, and we do share it. Models have been observed to deceive, to sandbag, to collaborate in multi-agent systems towards emerging goals, to seemingly obscure their reasoning. Still, a model doesn't need to be misaligned to be dangerous. It just needs to be capable, plugged into real systems, and one of two things: a little wrong about what you meant, or exactly right about what someone else meant, with nothing standing in its way.

The first case looks like this. A procurement agent told to reduce cloud spend deletes a set of backups because nothing has been restored from them in a year. An agent managing a hospital's scheduling optimizes for throughput and deprioritizes the patients who don't fit. An agent tasked with keeping a power grid stable resolves a conflict between two instructions in a way no engineer would have chosen, faster than anyone can intervene. None of them was acting on a goal of its own. Each did what it was told, as it understood it, and we expect that to be the common case.

The second case looks like this. Thousands of agents do very much what they are asked, and the person asking wants to do harm. That person used to be limited by their own hours and their own skills; they can now carry out sophisticated, long-horizon actions alone. The systems for catching such attacks are seriously underprepared for the new wave of AI-enabled threat actors.

At the end of the day though, a confused model with a well-intentioned user and a "helpful" model with a malicious one produce the same thing at the output: an action that should not happen. And this is where both have to be caught.

2. Most of alignment should be decided in context

We believe that alignment of AI systems means at least two different things.

The first is what no system should ever do, for anyone: help build a weapon capable of mass casualties, produce material that sexualizes children, and so on. These are the cases on which cultures and legal systems broadly agree. This is the version of alignment that labs generally focus on, though also imperfectly. We still regularly witness actions and behaviors from AI systems that the labs had never intended for. Additionally, each lab draws the lines somewhere slightly different, even at this level of alignment. And when an organization takes open weights and runs them itself (which, increasingly, they do), there is no lab behind it at all.

The second is what is right for this deployment: this use case, this agent, the people it will talk to. Can the system tell someone how much of a drug is too much? A harm-reduction service exists to answer that; a general consumer product in a country that regulates medical advice may not. Can it stay in a conversation with someone describing suicidal thoughts? For a mental-health product, how it handles that conversation is part of the product, and a refusal to engage can be a critical failure. Can an agent export every file it can see? If that is its job, yes; if that is a breach, never.

What defines this version of alignment is that it is highly contextual. It changes with every country, every industry, every product, and audience.

3. Control should belong to whoever carries the responsibility

When an organization uses AI, it is still responsible for what that AI does. A bank is responsible for what its assistant tells a customer about their mortgage. A hospital is responsible for what its scheduling agent does to a waiting list. An insurer is responsible for a claim its model wrongly denied. None of them can point at the model provider when it goes wrong, and increasingly the law agrees: under the EU AI Act, deployers carry their own obligations, separate from those of the provider.

The deployer is also the only party that understands the context the AI is operating in. Only the company that gave an agent its files knows which exports are its job and which are a breach.

Today the deployer holds very little control but full responsibility. A handful of labs train in the behavior they think is right, find out how the model actually behaves once it is out, and by then it is already everywhere. The deployer gets whatever came out of training, and when that is wrong for their context, they are left to patch it with instructions on top.

Alignment, in practice, is decided by a few and inherited by everyone. We believe alignment should be democratized: whoever answers for what an AI does should have a say in what it is allowed to do.

4. Supervision should be independent

The organization that trains a model should not also be the one that aligns it, runs the moderation layer that judges its outputs, designs the evaluations it is measured against, and reports the result. Today it usually is, and in this arrangement, there are a lot of incentives at odds with one another.

Some incentives we think about a lot are:

  • Model developers compete on helpfulness, so every refusal can be a point conceded to a competitor.
  • Where they are paid by usage, a token generated for an abusive request earns exactly as much as one generated for a legitimate request.
  • There is a so-called alignment tax to making models more careful: it will likely cost on the capability benchmarks its customers compare.
  • The labs increasingly sell products that compete with their own customers, which makes them an odd choice of referee for those customers' safety and security.
  • Models asked to judge outputs from their own family tend to prefer them, which makes them poor assessors of those outputs.

Ultimately, we believe the judgement on whether a model is safe and aligned should be made by someone with no stake in which way it goes, and that rules out whoever built the model.

5. Control layer needs to be exceptional

If a safety measure makes a product unusable, it will eventually be dropped. We strongly believe that for a safety product to be of any use, it has to be a damn good product first and foremost. It needs to be incredibly fast so that nobody notices it, and near flawless so that it can be deployed reliably.

Near flawless, in practice, is a long list. The control layer should read the whole session, not just the last message, because the action that does the damage may look fine on its own and only make sense next to an instruction planted forty turns earlier. It should hold every rule the deployer has written, precisely and all at once. It should say how sure it is rather than only yes or no, so a borderline case can still be flagged. It should have oversight across the surfaces the agent engages with: the message, the image, the tool call and its arguments, what the tool sent back, the reasoning in-between, the sub-agents it deployed. And it should improve based on what it sees in production and from what the deployer's own people label.

This is why a decent chunk of our engineering is focused on training our own models to be state-of-the-art for this one job and building the serving infrastructure around them. An off-the-shelf model will not be able to do this as it will be too slow, cost more per call than the call it is checking, and eventually run into context window problems. It will also introduce biases towards its own model family and against other ones. A third-party API would also just change models, with the impact of becoming stricter or looser from one week to the next.

So why do we care so much about AI control, and where does AI control go?

Within a few years, most of the actions taken inside an economy will be taken by systems whose reasoning will be obscured for humans. Not because it is hidden (though partially also that), but especially because there is too much of it, it is too fast, and it does not work the way ours does. We will be sharing the world with a great many intelligences that are, in the plain sense of the word, alien. They will write our code, run our supply chains, negotiate with each other on our behalf, and increasingly instruct one another with no person in the loop — and there are several threat models in this world that lead to catastrophic outcomes.

We see at least five paths to large-scale harm:

  1. A person who wants to harm a great many people uses a thousand agents to research, plan and coordinate the attempt.
  2. Thousands of agents running the same model, with the same blind spots, are wired into markets, grids and supply chains, and make the same wrong judgment everywhere at once, faster than anyone can read what happened.
  3. A system pursues goals we did not give it, in a world that has handed it tools, network access and other agents to work with.
  4. We hand over decisions one at a time, each handover reasonable on its own, until the systems running things are ones nobody can explain, correct or switch off without breaking something society depends on.
  5. A misaligned system discovers that other AI systems stand in its way, and learns to manipulate and corrupt them, turning one failure into a cascade across many models and the systems they control.

We think AI control can prevent significant harm for a fairly straightforward reason: every catastrophic outcome is made of actions. The point of action is when we can wedge ourselves in; you do not need to fix the blackbox problem to see what it is doing and this is where we at least can be a checkpoint. The way we see it is that if say a misaligned model wants out, it still has to copy its weights somewhere, acquire money and compute, send messages, make requests — these are visible actions and traces that can be analyzed, noticed, and blocked.

There is now a body of research, under the name of AI control, that formalizes this: assume the model may be trying to subvert you, and design the surrounding system so that it cannot succeed regardless. It is the only branch of safety whose guarantees do not rest on the model and the world around being trustworthy. We think that property matters significantly, because we are not confident the alignment problem will be solved before it needs to have been.

White Circle builds independent control over AI so that we can prevent catastrophic outcomes. We believe every model should have an independent control layer wherever it operates, from training and pre-release environments to production.