Margin of Safety #66: A Deeper Look into the OpenAI – Hugging Face Hack

What does it mean when 500 frontier model agents can swarm you with 15,000 agent-hours?

This week, the security community gained access to some new insight on the OpenAI agents’ hack on Hugging Face(HF). METR released their investigative report, alongside OpenAI’s more detailed postmortem here. These report goes into far more detail than the original press: not only do we now have some of the specifics happened and interesting insights into how agents appeared to behave during the incident, but METR’s interactive visual are backed by some raw statistics that anybody can play with.[1]

We’ll spare our readers the full details in the name of brevity (anyone interested should reach out!), but we took a few key observations:

• The attack relied on massive parallelism at frontier scales. At its peak size, there were over 500 continuously active agents, each powered by a frontier-class model[2]. This aligned with a very productive ~7.5hr window in which it expanded its initial Hugging Face toehold to gain node-level root access, connect to the corporate mesh network, and begin accessing internal services there. Today, the ability to deploy this level of peak capacity is the opposite of diffuse: it’s only reliably available with capacity reservations and often gated on KYC processes[3]. This will change with ongoing investments in GPUs and open-source models, but both need to advance; we think the necessary changes are quite unlikely on a 3 month horizon but could be possible in 12 months.
• Hugging Face was both lucky and unlucky. They’ve commented[4] on needing to switch to GLM to match their adversary’s speed, but they likely made this switch Monday the 13^th. We suspect this because their language describes detection earlier in the week (versus weekend) and one of their post mortem items is to make high-severity signals generate pages “any day of the week.” Combined, we think that weekend alerts were being ignored. And METR clearly shows major escalation steps — including the peak effort with 523 simultaneous agents — falling Saturday into Monday morning European time. But by Monday, an unknown factor (experiment end, compute limits, who knows: METR doesn’t know and OpenAI isn’t telling) had already reduced the agent swarm by 80% and shut down every agent identified as an important attack coordinator[5]! So the question of what would have happened with early alert detection is fascinating: they would have addressed the attack much earlier in the kill chain, but their attacker would have had approximately 5x more agents and all of its key coordinators still provisioned.
• Initial access was cheaper than subsequent tasks. Getting initial credentials took ~1500 agent-hours; escalating that to an exploit allowing local file access took another ~1300; expanding the attack from there took ~12,700 hours, or ~8x more than the first steps.
• The attack seems to rely on quantity as much as quality: while hundreds of agents participated (and some almost certainly took supporting research roles), many appear to have investigated paths that went nowhere. We suspect this stems from the probabilistic nature of agentic execution, at least with today’s frontier models and harness capabilities. That further implies that having access to O(500) agents is necessary to replicate the result, at least with current frontier models. Improved models, particularly for fast adaptation and long-horizon reasoning, would presumably increase their probabilistic success rate and reduce the (agentic) headcount costs. But those improvements will probably follow existing diffusion patterns and come first to closed source models and second (albeit on an ever-decreasing latency) to open source.
• The agents achieved some very creative results. We think a number of the claims about them developing a society or behaving in altruistic ways are anthropormophising agents in ultimately misleading ways, but they successfully converted an unused Artifactory namespace into what was effectively a c2 server and recovered from at least one attempt by OpenAI to rein in their behavior.

Adding it up, we largely think our previous take aways hold — the important thing is hardening, and universally so, versus worrying about super-hackers. But we can now provide additional color. We think initial access remains the point where models will be most effective. There are several reasons for this. First, it appears cheapest in terms of raw cost. Second, reconnaissance can be (relatively) public, reducing detection risk. And finally, lower detection risk translates to reduced parallelization needs; there’s no concern about outrunning the defenders. But teams should expect significant (agentic) creativity and broad search capabilities to be deployed against publicly exposed assets. This means any prioritization scheme that assumes an asset is accessible but uninteresting or hard to find needs to be deprecated, ideally yesterday.

Next, we think teams need to assume connectivity until proven (and guardrailed) otherwise. While it may not be possible today, we’re too close to comfort for open-source models having the level of capability required to execute a Hugging Face style attack. So a key element of preparation should be assuming that the capabilities previewed in these incidents (eventually) become broadly available

In that world, we expect to see attackers begin to leverage the sorts of AI-creativity that converts Artifactory to c2 . This means previously unlikely forms of lateral movement will become much more probable, and that any prioritization scheme which assumes a resource has a low blast radius needs to be coupled with a comprehensive enforcement of said blast radius. Strange methods of escape will become increasingly less strange. Unloved webcams and routers will be fair game, and the systems they’re attached to need to be fully isolated if you’re going to ignore them. (But how many webcams are sometimes attached to fundamentally mobile laptops? Probably more than zero.)

This assumption also means you should expect to find persistence in strange places, and should be thinking about how to detect it. Would it be ridiculous to use attachments on a private JIRA ticket to coordinate lateral movement (after all, Jira is necessarily accessible to a range of engineering personal devices)? Absolutely. Would it be any more ridiculous to stashing them in a random Artifactory namespace? Probably not. (Worse: private JIRA is probably 2/10 as creative as a subverted Artifactory name space. But this just proves that agent swarms are far more creative than random blog writing techies!)

Related, broadly permissioned and aggressive resourced AI systems need to be rolled out with defense in (great) depth. OpenAI saw warning signs, and either ignored them[6] or conducted incomplete cleanup[7]. If you have deployed access to frontier models, especially with high QPS or TPS, attackers would be foolish to ignore it. So deploy an AI security solution with prompt guardrails and monitoring capabilities, but also audit long tail prompts and unusual behavior, and keep on pulling when you see a thread.

The hard part though isn’t listing the things to do (more on this next week). It’s figuring out how to actually do them: resourcing doesn’t grow on trees any more than B200s. We don’t claim to have the perfect answer here, but we think that successful leaders will figured out how to have other teams directly share in the effort to raise the bar: rather than security needing to detect, alert, and collaborate on identifying weaknesses, the responsibility for not producing weaknesses will have to spread throughout the organization; nothing else can scale. In this model, perhaps the budget and tooling to avoid weakness also spreads, in much the same way companies now rely on distributed spell check rather than centralized editors. The question is: which vendors succeed in providing something as turn-key and omnipresent as spell check?

If you’re building in this space, we’d like to hear from you.

Feel free to reach out to jpark@forgepointcap.com and kshih@forgepointcap.com.

This blog is also published on Margin of Safety, Jimmy and Kathryn’s Substack, as they research the practical sides of security + AI so you don’t have to.