Margin of Safety #71: Vuln-zero might not mean what you hope
Jimmy Park, Kathryn Shih
October 27, 2026
- Blog Post
AI may eliminate many traditional coding vulnerabilities but that is far from vuln-zero
Last week, Jason Clinton of Anthropic warned that while (very) rough times are ahead, we’ll eventually get to a vuln-zero world when AI-powered code analysis has ferreted out the world’s accumulated software vulnerabilities and commoditized small model capability prevents future introduction (see his tweet here). While we agree with much of his premise – rough times are, sadly, ahead, but many classic sources of vulnerabilities will massively shrink in the medium-term – we disagree with his conclusion. We think the CISO of 2028 may well have extra grey hairs, but he or she will still be forced to worry about software quarantine and updates.
There are three reasons for this: first, we expect that developers will sometimes take ill-advised shortcuts and bypass safety tooling to build and launch preventable vulnerabilities. Second, vulnerable software can arise from malicious hijacking as well as implementation error, as in the SolarWinds case, and the result has significant response overlap; safely removing or updating a highly problematic installation. Finally, there are classes of software weakness that stem not from an incorrect spec implementation (the vulnerabilities now being discovered at scale), but rather from a spec that contained a fundamental weakness. Not only have we seen less evidence that AI successfully finds those weaknesses, but there’s reason to suspect that the integration of LLMs into the software stack will increase the error frequency.
We think the first two points don’t need much explanation. If it’s possible to YOLO software into the world, some people (and teams) will likely feel the temptation to do so, especially in projects that seem (initially) smaller scale[1]. The open question as we see it is not whether people will succumb to the temptation, but rather how far industry will go in hardening against it. For example, in extreme conditions, we can imagine a retreat from existing Linux flavors and a move towards more locked-down, commercial-OS style environments where all packages must be signed by a central authority that also enforces basic security scanning. Vendors providing hardened FOSS images and packages are already providing variations on this capability but primarily focused on server versus desktop images. But per our second point, even these cases are vulnerable to a supply chain compromise in which the vendor responsible for distribution is attacked and their infrastructure used to distribute malicious content. We think CISOs will find it hard to fully outsource this issue of trust, particularly for closed-source software.
For the third point, we think two categories of vulnerability will certainly persist and probably grow. The first is from bad specs. At a high level, bad specs often exist when commercial pressures favor lower-friction solutions; simpler sharing controls, less restrictive access controls, etc. This is a classic issue that predates LLMs. Examples run the gamut, but include:
- Zoom using a local webserver and questionable security model to support maximally convenient (but minimally secure) meeting auto-join behavior (here – Zoom has since corrected/removed the behavior). Our inner cynics compel us to point out that Zoom’s insecure behavior from 2019 is exactly the kind of workaround Claude Code enjoys cooking up in 2026.
- Strava leaking non-public military locations, because observers could monitor deployed troops’ training runs.
- Vendors who intentionally choose more permissive account safety settings – lots of password retries, optional MFA, etc – because it encourages product adoption when accounts are easy to set up and access is hard to lose (for one study demonstrating the phenomenon, see here).
- Cross-tenant data leakage on OpenClaw (here), stemming from its design decision to keep things simple[2] and assume local deployments with a trusted user (and one such user per gateway).
Across these examples though, software weaknesses didn’t come from an engineering oversight or implementation error (though OpenClaw also did have several serious CVEs). Instead, each stemmed from the intentional specification of permissive behavior, and that behavior being a bad fit for the ultimate real-world environments in which the software systems found themselves. Ideally, vendors would correctly anticipate and threat model all the real-world environments in which their products will be used. But this is exactly the kind of behavior that we see less of in a vibe-coded world, and it doesn’t fit into a Claude Code or Codex tool on a developer’s endpoint. After all, it needs alignment between the team building the tool and the team that’s going to be marketing and selling it, along with a commitment to revisit the requirements and downstream design choices if use cases expand in the future.
Even if you solve for this though, there’s a more direct issue posed by LLMs. Moving past AI for coding, most modern systems also directly invoke LLMs, and that means risking prompt injection. But prompt injections stem from a core architectural gotcha in current models: within a prompt, control plane and data plane are conjoined, and we have not yet devised a way to fully separate them and prevent rogue data from affecting the control plane[3]. As a result, defenses against prompt injection are either structural (adjusting the spec so that the model output is used in more constrained ways, but typically at the cost of system capability[4]) or probabilistic (attempting to detect and block all possible injections). The first requires exactly the sort of tradeoff that we think leads to spec level bugs, and the second requires ongoing patching as new corner cases are discovered and as updates to the probabilistic guardrails are required.
Combine these factors and we expect to see software vulnerabilities continue, but for them to move up the stack, metaphorically speaking. We expect fewer memory safety errors, and more issues stemming from bad spec choices or model guardrails failing in the fact of new and emergent task variations. Long run, this probably does mean fewer total vulnerabilities – after all, you can cram a lot of dangerous pointer hijinks into a very small codebase – but it also means that the vulnerabilities we’ll see will shift towards more complex fixes and deployments, since high level issues often require a spec and thus behavioral change[5].
So what’s a forward-looking practitioner to do? First, don’t get too eager to free up vulnerability budget in 2028. But more usefully, take a slightly broader lens on vulnerability readiness. For both internal teams and vendors, vulnerability management will go beyond having the best coding tools to prevent classic errors[6]. It will require taking a thoughtful, well documented approach to threat modeling, ensuring that signing and distribution infrastructure is locked down, recognizing embedded models and guardrails as patch candidates, and having a plan in place to make painful spec changes if or when it’s warranted. Enforcing this on internal teams will be hard; enforcing it on vendors will be harder yet. But for folks with their eyes on the horizon, we think the current vulnerability wave gives a unique opportunity to press vendors on hardening; not just against things like memory safety, but also for forward-looking security.
If you’re building in this space, we’d like to hear from you.
Feel free to reach out to jpark@forgepointcap.com and kshih@forgepointcap.com.
This blog is also published on Margin of Safety, Jimmy and Kathryn’s Substack, as they research the practical sides of security + AI so you don’t have to.
[1] If anyone doubts this as an occasional human tendency, we semi-seriously recommend checking out https://www.reddit.com/r/Decks/ for evidence that some people will ignore best practices even when their literal neck is on the line.
[2] Keeping things simple is often an engineering best practice: https://en.wikipedia.org/wiki/KISS_principle so we find this example extra illustrative. How often do bigger projects grow from small prototypes that may have pragmatically taken simplifying assumptions at the spec level?
[3] Vulnerability guidance aligns with this. Per MITRE, prompt injections are an application-level weakness: https://cwe.mitre.org/data/definitions/1427.html. But if there’s one trend that we feel the anecdata supports, it’s a failure of many vendors to commit to the degree of spec-level tradeoff required to fully prevent these issues.
[4] Tl/dr: if your spec combines access to private data, exposure to untrusted content, and the ability to send data to untrusted parties, you’re going to have a bad time. But (for example), this is core to most personal assistant type tasks. If you want to read more, there’s good practical coverage at https://ai.meta.com/blog/practical-ai-agent-security/
[5]Sure, fixing memory management or command injection technically products a behavioral change (stop doing the vulnerable thing), but many fixes that involve changing a spec involve very user-visible impact; often closing the door on an existing benign flow.
[6] We aren’t including steps to prevent YOLO’ing code into production as a future requirement, because we’re assuming it’s a classic error!