Margin of Safety #70: More Maximalist Anthropic Marketing on ‘AI on IoT’

Residential slime molds don’t pass the sniff test, and existing controls can protect the enterprise

Another month, another piece of maximalist cybersecurity marketing. Last week, Anthropic published a blog speculating that self-replicating AI botnets which shard smaller models against a dozen+ IoT devices could be a near term threat. While these concerns have a kernel of truth – Anthropic has demonstrated that the sharding is possible – we think that the real threat is lower than suggested for both technical and practical reasons.

We’ll get into both sides of our non-concern, but let’s start with the technical. The blog somewhat interchangeably references IoT devices and Android TV class devices. But IoT devices run the hardware beefiness gamut. In practice, to be a useful part of a slime mold, an IoT device needs to have enough memory to hold its core OS, about .5GB of model weights, and then run some inference on top. It also needs the memory throughput and CPU heft to operate on that data. Add in the existing OS, and we think this translates to the device needing about 2GB of physical RAM (but we’d revise downward with evidence; if you have it, contact us!)

When we go through the world’s IoT devices, we see that many of them fall below that memory bar. The ones that meet it are largely streaming and video conference hardware (plus one fridge that Samsung has kitted out with spare Smart TV parts[1]. Sadly, Claude has declined to do more targeted math on memory throughput[2], which would provide a second dimension on which some hardware may be disqualified, but even the research it’s willing to do is illustrative.

We’ve labeled popular samples from a few categories to help build intuition. For interactive chart – check out this link.

Even though it cuts out most devices, we think the 2GB cutoff may still be optimistic. For example, the Apple Watch has a good chunk of RAM[3] and is included. But the watch is a lot of (digital) punch packed into a tiny box with a finite battery. Spend too much power shoving bits over the network and/or running matrix calculations, and users will not only notice the battery dying, but they might also notice their wrist heating up. So, while the watch technically meets our memory cutoff, a wise slime mold might avoid it.[4]

No matter how you slice it though, useful-to-inference IoT devices are a fraction of total household devices and centered on a low number of categories like connected TV and potentially vacuums/lawnmowers[5]. The 2022 US average was ~4 smart/connected TV devices[6] per household, including everything from consoles[7] to sticks like Roku. Even assuming some subsequent growth, we think few households have 14+ high end IoT devices. This means molds must span households. We lack hard numbers but wouldn’t be surprised if it’s 4+ (at least until we see more evidence supporting the offensive efficacy of small models). But spanning introduces the risk flagged in the blog: ISP-level detection and containment. As soon as the ISP disconnects one node, inference stalls until the worm infects (or promotes) a replacement node.

Worse, the worm needs to persist without device owners noticing and returning their devices. Commodity-priced Android TV boxes lack the memory for inference to be co-resident with primary workloads like smooth video streaming (unless vendors start overprovisioning by a couple GB, but with today’s memory prices? We think not!) so an effective worm must sit on disk and only run inference when users are away. The more households and device types you join in, the harder this balancing act becomes; at some point, it collapses[8]. The only place where it’s potentially less of an issue is the enterprise, but more on that later.

Beyond the hardware constraints, there’s also a practicality concern. Small models like Gemini Nano are already being shipped on flagship Android phones, and we expect other Android class devices to follow. So if this attack ever works with truly small models, a hypothetical AI worm could mount a living off the land attack and use existing models rather than replicating its full stack (which would contend with existing ones for memory anyways!). We think that this is a bigger threat as IoT hardware improves. Local AI is likely to be a motivating driver of any such improvement, and the more local AI there is, the less it needs to be part of the payload.

These household level impracticalities leave us with conviction that corporate environments are the hardware sweet spot. Not only are Jetson-class chips found in enterprise devices, but enterprises are also (relatively) full of high-end video conferencing devices – sharp eyed readers will notice those clustering at the top of our chart[9]. There can also be a lot of these devices on a subnet. So unlike households, corporate settings might have critical mass accessible devices, and employee work schedules can cause them to be simultaneously available all night. But that’s assuming you can get in: unlicensed Android is probably one of the weakest environments within the higher-powered device classes. Enterprise, flagship Android, and Apple devices are all far more likely to use secure boot and have basic hardening measures in place.

Even if we assume that these more-hardened devices are within the capability range of the hypothetical worm’s model though, we still have our next question: how does this attack actually occur? Anthropic’s scenario was a rogue agent effectively replicating for the misaligned lulz. We can’t disprove ‘for the lulz’ as a possibility, and in fact we think it’s the only viable one. Traditional attacker motives –economic or geopolitical gain – don’t lead to this style of self-replication. Assume, for the sake of argument, that an attacker could gain persistence within some pool of highly capable IoT hardware. At that point, why not deploy ransomware? Or skip the installation step that an OpenAI employee recently tried and just capture and exfiltrate video calls (or, if you’re shipping models, classifying calls and only extracting the juicy bits)? Traditional attackers are too pragmatic to let an AI slime mold risk detection while bumbling around a high value network when there are so many better things to do.

This leaves the misaligned agent. We’re the first to agree that, given recent news, a foundation lab ideal for a model to escape containment. But an agent successfully assimilating OpenAI’s video conferencing hardware is OpenAI’s problem. It becomes a problem for industry when it expands into other enterprises. But responsible enterprises should already be protected ISPs aren’t the only player with detection capabilities; network owners in general are similarly empowered. Enterprises should be broadly concerned about the potential for classic adversaries to subvert their IoT hardware, including video conferencing hardware. And conveniently, the steps that prevent and detect classic subversion – rapid patching schedules, ITDR coverage, network segmentation, etc – will also be effective at preventing any hypothetical slime mold.

So once again, we aren’t losing sleep about frontier AI attacks relative to the adjacent and less sexy issue: there are a growing number of increasingly capable IoT devices. They universally need diligent patching and monitoring: enterprises should take note, because economically motivated attackers eventually will. Beyond that, many consumer devices are Android-based, producing a doubly bad patching situation: even if owners were good at patching their devices (which they aren’t), Android vendors, especially unlicensed ones, are bad at offering patch maintenance for the extended lifetime of an IoT device. So, while we don’t expect to see self-replicating LLMs soon, we sadly do expect more IoT botnets as the vulnerability tsunami comes for that space. If you’re responsible for enterprise environments full of modern, high powered IoT devices, you should recognize that these units are approaching fully fledged endpoints in their associated capabilities and risks. And if you’re a consumer of IoT devices, now would be a great time to start thinking about how much you trust vendors of your higher-powered devices to offer long term patching, hopefully on an automatic basis.

If you’re building in this space, we’d like to hear from you.

Feel free to reach out to jpark@forgepointcap.com and kshih@forgepointcap.com.

This blog is also published on Margin of Safety, Jimmy and Kathryn’s Substack, as they research the practical sides of security + AI so you don’t have to.