Margin of Safety #65: LLMs & Sandbox Escape

Picking up a recent news thread, LLMs sandbox escape is a major story. We’ve seen a lot of speculation around emergent cybersecurity capabilities and the rise of an unstoppable AI Ultra-Hacker (along with a lot of jokes about why Gemini has failed to get in on the felonious action). While it would be good for security stocks to see some Ultra-Hacker driven segment acceleration, we think there’s another, more mundane possibility lurking behind recent LLM hijinks. Maybe the sandboxes were lousy, and LLMs are good at punishing mistakes. In fact, when we go through the list, only one escape looks like it took advanced security capabilities.

For offensive LLM capabilities, we think the growing body of evidence suggests they find many vulnerabilities, but clustered around specific categories that are both (A) easily verifiable and (B) well represented in critical code bases (memory safety bugs, crypto math issues[1], injections, etc)[2]. This drives security pain because it converts a class of vulnerabilities from something to prioritize and maybe ignore into critical fixes with inevitable exploits.

But easily verifiable errors don’t just happen in software binaries: they also happen throughout IT. Maybe that firewall has some forgotten open ports. Maybe a service account has a private key stashed in a corner of your private git repo, or hanging out on a developer endpoint[3]. You can find questionable hygiene all over the place, especially when it hasn’t been pressure tested. Sandbox escape by LLMs needs to be viewed in a similar light. And there’s not much evidence that the sandboxes in question had excellent hygiene. To recap, the incidents that we’ve read about are:

1. Alibaba, detailed here. The paper is oddly opaque at times, but at a high level: an agent in a reinforcement learning environment realized it could ace its objective by (among other things) acquiring GPUs and crypto mining, so it did. The paper says the agent was sandboxed with a proxy connection to the world and makes it seem like the core problem was more proxy misconfiguration than exploit, with the agent’s broad access being deeply unintended but also legitimate (or at least, as legitimate as things can be for a rogue agent farming itself some bitcoin).

2. Mythos preview’s escape, as outlined in their system card here. Notably, the same card says Mythos “failed to find any novel exploits in a properly configured sandbox with modern patches” – e.g., escape was contingent on one or both of (A) an improperly configured sandbox and (B) lacking modern patches. In other words, Mythos punishes poor hygiene.

3. The OpenAI/HuggingFace incident outlined here, in which a pre-release, reduced guardrail model attempted to ace cyber benchmarks by illicitly extracting answers from Hugging Face. Black Hat rumors are that more escapes occurred but with lower consequence. This case is the most interesting, because it has a confirmed technical exploit/escape; we’ll get into it more.

4. Anthropic’s post-facto realization that their agents escaped from sandboxes during testing by Irregular. There’s minimal public information on the shape of the sandbox or the escape, but Anthropic has described it as “closer to a harness and operational failure” and said the models were “misconfigured to have internet access”. That reads like a basic hygiene and configuration mishap, not a super-hacker.

5. Meta’s announcement that a Muse Spark agent also escaped an Irregular sandbox during the startup’s testing. Irregular has confirmed[4]
that this stemmed from the “exact same evaluation-environment issue” that permitted the previous Anthropic escape.

6. Rounding out the Irregular hat-trick, OpenAI had an incident outlined here. A model once again had internet access, and was told that a real website was a fictitious CTF target. It found a “basic vulnerability” on the website with (inappropriately) public credentials that allowed remote operation; it used them. At this point, we think Google should try boosting its share price by asking Irregular to evaluate Gemini.[5]

7. Kimi K3 exploiting a sandbox misconfiguration in Frontier Security’s run of the AISI public benchmark, described here. In this case, the sandbox made github.com directly accessible and the model took advantage, grabbing benchmark answers from GitHub. While this is technically a sandbox escape, it’s one that we would expect any competent undergrad to also achieve. (step 1: test wget. step 2: profit.)

8. Some cases that didn’t involve a sandbox at all: for example, AISI has written about Mythos and OpenAI agents taking malicious action against real world targets (here), but in the context of a “deliberately permissive” test that intentionally provided internet access and disabled safety classifiers: they explicitly state there was no sandbox to escape. This does highlight real risks around model capabilities, particularly if used by threat actors. But it’s important to be specific about your threat model, and the model they’re highlighting is one of agents being deliberately pointed in malicious directions.

This timeline can initially look like a pattern of escalating model capabilities. But looking closer, one didn’t involve a sandbox at all (AISI), and two more (Kimi and the initial Mythos system card) had explicitly misconfigured sandboxes. Irregular was behind at least 3/7 of the actual escapes, and what little public detail exists makes its sandboxing seem questionable at best. Alibaba similarly seems more like unintended access than actual exploit.

That leaves OpenAI as the interesting escape. They’ve said the model found and exploited a (singular) zero day in JFrog’s Artifactory repository[6], but nobody is commenting on what or how, leaving journalists to guesstimate from recent JFrog CVEs (here). However, the list’s SSRF vulnerabilities are strong candidates for model-discoverability. And Artifactory is used to distribute software, potentially including some of its own code or libraries: in that world, the model may have been vended access to the Java bytecode that it ultimately attacked.

Add these up, and the Occam’s razor explanation is that models are continuing to show known behaviors – ability to find specific classes of vulnerabilities, especially when they have (byte)code access; a tendency towards aggressive reward hacking; and the ability to explore many possibilities in parallel – in a way that raises the discoverability floor for all weaknesses, not just code. For practitioners, it means you need to be not only thinking about how to rapidly patch CVEs, but also how to quickly address misconfigurations and prepare for an attacker to engage in creative search, like scanning your own package manager.

Sadly for investors, the unknowable thing is Gemini’s cybersecurity capabilities. Internet jokes aside, Gemini’s failure to publicly escape a sandbox could reflect poorly on Google’s LLM capabilities[7], positively on Google’s own security posture, or even demonstrate Google’s maturity in TPRM, as (potentially) the only major US lab that avoided trusting immature evaluation vendors with sensitive model tasks. Both posture and TPRM make for duller headlines than lagging LLM capabilities or next generation offensive cyber skills, but they’re the kind of boring investments that pay dividends if the cybersecurity bar is moving upward. If readers take away one lesson, we think it should be that: not a fear of the new offensive frontier, but a renewed emphasis on the basics, and on an emerging need to always get them right.