Margin of Safety #25: Security Automation and Monocultures

TL;DR: In security automation, unpredictability is a defender’s best weapon against attacker modeling and monoculture risks

*In this week’s blog, we worked with Shane from our team so we’ve 10x’ed the insight.

Now that everyone is marketing a security agent and, in our eyes, starting to market attack path or kill-chain based prioritization, we think it’s a good time for defenders to consider the impact of monocultures in adversarial contexts. At a high level, we believe it’s important that defenders have a plan to make some strategic portions of their strategy difficult or impossible for attackers to predict. But how does that square with a move towards outsourcing increasing numbers of higher-level decisions to automated solutions? And does the situation become particularly dire if many products are built on top of the same handful of AI models, and thus share the same decision-making tendencies (which may even be feasible for attackers to manipulate)?

We think that the answer for defenders is to ensure a small handful of key defensive decisions are difficult for attackers to predict, either (A) because they remain in the hands of human operators *or* (B) because they rely on solutions which are sufficiently customizable that a savvy attacker cannot predict their responses in the context of a given environment. Let’s dive in.

(Source: Star Wars: Episode IV – A New Hope)

Why variability is important in adversarial contexts?

At a high level, variability prevents attackers from easily adapting to defender techniques. Examples range from immune systems (where things like banana blight can easily affect monocultures) to the advantages of randomized algorithms against pathological cases. Particularly when the adversary is smart – like in cybersecurity – having a completely fixed procedure allows a determined adversary to predict and war game your defenses, selecting actions that will avoid tripping them. We already see this in some contexts in security, such as EDR evasion. In abstract, EDR evasion becomes much easier when there are a dozen vendors versus 200: it’s much easier for an attacker to verify that their techniques work on the small number of available solutions.

Where is unpredictability important in security?

First, let’s talk about where unpredictability isn’t important. Some things in security have a clear right answer, and that answer doesn’t leak anything to an attacker. Best practices are one example. Yes, you should use MFA; there shouldn’t be any randomness in that decision. Another example is tasks like alert enrichment or fact gathering; for a given context, there is a correct answer to most of the enrichment tasks you can perform, and the fact you know your environment well enough to perform that enrichment should not be a surprise to attackers.

Things get more interesting when cost and tradeoffs enter the picture. If enrichment is expensive and only some alerts qualify, a clever attacker that knows this strategy might focus on activities which generate alerts that fall just below that cutoff and never get augmented with the additional context which would facilitate detection. Similar dynamics could potentially arise in alert triage: in a world where only a small fraction of alerts are investigated, an attacker who can fully predict prioritization will be able to fly under the radar indefinitely. This dynamic points at the key consideration for predictability: we think CISOs should attempt to ensure that the decision points which most impact attacker detection are difficult for attackers to predict or wargame.

How does a CISO do this?

We think the key moments for unpredictability are in deciding what to ignore. These are the decisions attackers can obviously influence or game to sneak past defenses, and one in which AI models increasingly play a role (at least in some SOCs). If, as an attacker, you can clearly predict the types of detections that will be ignored, you can likely calibrate your actions to align. Conversely, as a defender, we believe CISOs should be pre-empting the issue by considering what element of their detection or prioritization stack is not subject to gaming by an attacker that’s performed basic reconnaissance (or gained an initial toe-hold, with appropriate levels of visibility, into an organization). Deeply contextual prioritization algorithms could satisfy this requirement, but shallow ones likely will not; they’re unlikely to use sufficiently subtle signals to be a surprise to anyone, including an attacker.

We also believe CISOs should test LLM-powered solutions against resistance to indirect prompt injections. Can an attacker emit a stream of log entries that convince an AI analyst newApp.exe is a harmless engineering network test? Hopefully not, but CISOs should do more than test.

How can tool creators do this?

If you’re the person creating the tools a CISO will use, we think there are a few strategies that can be used to reduce problems of predictability. In no particular order, three that we like are:

1. Highly contextualized or randomized algorithms: One solution, alluded to above, is to make things like prioritization algorithms use enough contextual data that an attacker cannot reasonably predict them in a useful fashion. The challenge is finding enough bits of information that can and should usefully improve the decision while being difficult for an adversary to game or work around. For non-prioritization algorithms, randomization may be an option – it’s been historically used in techniques ranging from ASLR to honeypot deployments – but it’s often more applicable to topics such as resource placement versus effort allocation.

2. Human input for key decisions: The most obvious solution is to allow humans to make the key decisions. Taken directly, this might seem like it can undermine the efficiency gains that CISOs are hoping to bring to their operations. However, another way to achieve human input is to allow the capture of fine-grained human preferences, which are reused across many decisions. You can think of classic SOC detection rules as one flavor of this preference capture – those rules are hard to predict in their entirety and define the set of things that are worth looking at. Any sort of complex setting or preference tuned model can also check this box; the challenge is making sure that the expressed preferences are both valuable and sufficiently nuanced as to add unpredictability.

3. Reduce the unobserved surface area: Gaming prioritization is an issue because there are a class of alerts or actions that a security team would ideally take, even for low priority issues, but is forced to ignore due to resource constraints. If you can make a process so efficient that everything gets attention, then there’s nothing left to game. Obviously (to us at least) nobody will fully achieve this result, but we think it’s worth calling out to at least partially sidestep the risks of a predictable system.

Conclusion / Takeaways

In conclusion, as the security industry moves toward automation and increasingly uniform model-driven decision-making, defenders must resist the pull of monocultures. Attackers thrive when they can predict responses, and widespread reliance on a small set of shared reasoning frameworks only amplifies this risk. CISOs should deliberately preserve variability in the most attacker-relevant decision points—especially in what gets ignored—through human judgment or sufficiently customizable systems that cannot be easily gamed. By embedding unpredictability into their defensive posture, organizations can make it harder for adversaries to reliably anticipate outcomes, preserving a critical edge in an environment where predictability often equals vulnerability.

Reach out to us if you are building in this space. We have some thoughts!
Kathryn Shih – kshih@forgepointcap.com
Jimmy Park – jpark@forgepointcap.com

This blog is also published on Margin of Safety, Jimmy and Kathryn’s Substack, as they research the practical sides of security + AI so you don’t have to.