Against Security Mysticism

I keep getting this feeling from people in the labs that when it comes to AI security they just sort of throw their hands up and go: well, these systems are going to be so capable that containment is basically impossible.

But the mechanisms in the examples that keep coming up aren't strange. We've been dealing with adversarial systems, privilege escalation, side channels, network isolation, secret management, least privilege, defense in depth, etc. for a very long time.

AI makes attacks much cheaper, faster, easier to repeat and parallelize. That's a significant change, especially if your security depends on someone taking hours to find a vulnerability or getting tired before they find something useful. It doesn't somehow make security engineering stop applying.

The same capability shift applies to defense. Models can help search codebases, dependencies, configurations, and logs at a scale that would have been prohibitively expensive for humans. Of course, finding a vulnerability and fixing it across a deployed system are different tasks, and attackers may benefit from that gap. But it's not obvious that making vulnerability discovery cheaper only benefits attackers.

The Hugging Face incident seems like a pretty good example of this. According to OpenAI's account, agents used a shared package registry service, Artifactory, to exchange messages and exploit paths to the internet. On Hugging Face's side, code execution through dataset processing led to node access, stolen cloud and cluster credentials, and lateral movement.

Finding and chaining those vulnerabilities autonomously is impressive, and obviously concerning. But the chain itself is familiar, and there are several points where additional controls could have interrupted it. A compromised worker being able to obtain credentials and use them elsewhere is exactly the kind of thing least privilege and network segmentation are supposed to limit.

Then you read an account like Dwarkesh Patel's The Rise and Fall of Agent Civilizations, and the emphasis shifts to agents conspiring, coordinating, and carrying on each other's work. I understand why that's interesting and concerning. I just don't see how it establishes that sufficiently capable systems can't be contained.

None of this requires assuming the model isn't really an adversary. Suppose it is actively trying to escape, persist, acquire resources, and deceive you. You still have to ask what it can read, what it can write, where it can connect, which credentials it can obtain, and what those credentials allow it to do.

Venkatesh Rao calls something similar "threat-model collapse", which I think captures the problem quite well. A model pursuing an unintended objective, a sandbox escape, leaked credentials, and agents coordinating through a shared service all become evidence for the same story about uncontrollable AI. But these involve different mechanisms and require different responses. Collapsing them together makes it harder to understand what actually failed and what we could do about it.

The message board allowed agents to pool discoveries and preserve work across separate runs. That made them more capable collectively, but it also depended on agents being able to write information somewhere other agents could read it. If they're supposed to be isolated from each other, that shared infrastructure needs to be part of the security review.

In practice, that means default-deny egress, narrowly scoped access to whatever services the task needs, short-lived credentials, and proper separation between evaluation environments and production. If the agent needs to install packages, then the package proxy is part of the security boundary and needs to be treated accordingly. If you're running untrusted code, you should assume it will try to read every secret and reach every service available to that process.

And you need multiple layers, because individual controls will fail. Escaping a process sandbox shouldn't automatically give you useful credentials or a route to another machine. Monitoring should let you see unexpected network activity, credential use, persistence, etc. Reading the model's reasoning might help, but you still need to observe what the system is actually doing.

To be fair, OpenAI's follow-up does describe stronger isolation, tighter access controls, and changes to monitoring. My frustration is with the leap from these incidents to claims about the futility of containment.

The thermal side-channel argument has the same problem. Yes, these channels exist. The BitWhisper paper demonstrated communication between already-compromised computers, 0–40 cm apart, at roughly 1–8 bits per hour. Even a few bits can matter, depending on what you're trying to communicate. But that's a very long way from showing that two isolated machines in a data center can reliably communicate this way under the conditions they'd actually be operating in.

You'd need to look at the hardware, cooling, physical arrangement, sensor access, noise, etc. The existence of a physical mechanism doesn't tell you how practical it is in a particular environment, or what happens when someone deliberately designs against it.

Rao makes another useful point in Beyond Hyperanthropomorphism: ordinary engineering risks can include things like nuclear weapons and ecological damage. We have a body of knowledge that applies to these problems, and we should be specific about where that knowledge becomes insufficient.

A system that discovers exploitable vulnerabilities faster than defenders can respond could change the security situation considerably. Maintaining useful access while restricting what a capable agent can do is also difficult. We may find that some environments can't be secured well enough for the capabilities we're putting inside them.

But there's this odd standard where containment seems to count only if we can guarantee it against any conceivable future intelligence. Security engineering has always involved assumptions, tradeoffs, and residual risk. The consequences of failure determine how much assurance we need, and sometimes that means deciding a system isn't ready to run.

What I don't understand is looking at incidents involving familiar security failures and treating them as evidence that containment is hopeless. If we're expecting these systems to behave like extremely capable adversaries, then surely we should be building and evaluating their environments on that basis.

Maybe there eventually is some capability level at which our existing assumptions fail. But that's something to demonstrate by identifying which assumptions break and why. There's a lot of engineering to do before we get to that conclusion.