Margin of Safety #60: Open Source AI Is Back — And Nobody’s Hands Are Clean

Think Western AI is safe and Chinese AI is compromised? Think again… neither side is squeaky clean

Open source AI[1] is having a moment (again). Models like DeepSeek and GLM are once again appearing to close the performance gap with top western models, but this time in the macro context of exploding LLM bills and newfound cost concerns. So in many ways, the conversation is returning to where it was a year ago: can western CTOs trust Chinese models enough to achieve cost savings by swapping them in? We don’t know. But uncomfortably, we don’t know what’s in any LLM, regardless of provenance.

The best open source models are Chinese, and have been for some time. We believe this is because distilling from multiple frontier models is the winning strategy for producing a powerful open weight model[2] on a reasonable budget, and it requires operating outside US terms of service and IP law, something Meta and other western providers can’t match.

Regardless of how the open weight models are coming into existence, preliminary reports from their providers suggest they’re powerful. Z.ai’s initial benchmarking of their new 753B open weight model GLM-5.2 shows it edging out GPT-5.5 on the widely referenced FrontierSWE benchmark while requiring 1/6^th the inference costs. Real world performance is as-of-yet questionable, since even llama4 was initially purported to outperform, but there’s a real chance that GLM ends up as a performant and deeply cost-effective option in the space. This makes the enterprise value obvious: a massive decrease in cost, full control, and no vendor lock-in: what’s not to love? Even if it underperforms on some tasks, we know enterprises are throwing AI at many tasks that probably don’t require a full frontier model (eg, slide conversions). In that world, we can easily imagine a future where open source gets 80% of enterprise workloads and only the top 20% (or less) of tasks by complexity go to the highest cost options.

But this assumes no security concerns, and those are back in full force along with the interest in Chinese open weight models. Before getting into those, we need to be clear on llm versus harness versus app. The water continues to be muddied with loose naming in the space. For example, Claude is an LLM, but most people interact with claude-the-model when it is surrounded by a harness (claude code) or an app (claude.ai).

To get specific, there’s 3 layers of the stack we care about. Starting with the inner core and working outward, we get:

1. The core LLM. This is the model and the weights that run on the GPU. You give it some text, it gives you a response. It can’t do anything but that, which means it cannot (for example) access the local system or network to perform adversarial actions, even if everything is wide open from a security perspective.

2. The harness. This is all the code that sits around the LLM/core model to help it be useful for tasks. Think context management & memory, tools, and so on. This layer is where you start to see potential for malicious behavior; adversaries can ship malicious tools, smuggle things into the memory system that cause behavioral aberrations, and get up to various other forms of mischief. It’s possible to strictly sandbox a model + harness combination, but as the harness becomes more powerful, sandboxing becomes harder; after all, the promise of tools is that they can interact with the world. It’s hard to have it both ways: the more a tool can interact with the world, the riskier an untrusted tool is.

3. An application. This is a full, user-ready product that incorporates non-harness components like a user interface, business logic, licensing & payments, etc. Claude desktop is an application, as is the DeepSeek app. Applications have all the risk of a harness, and often more, since many of them have a SaaS component. Once you’re dealing with some of your data being sent to a remote provider, you only have as much data security as you do trust in the provider. This is why some enterprises find hosting DeepSeek perfectly safe – it’s just a raw LLM – but absolutely did not want their employees using the DeepSeek app, which stores and processes data in the PRC.

But all that doesn’t mean the raw LLM is without risk. The more subtle threats can operate through influence rather than exfiltration. A model trained to favor certain software packages or coding patterns could steer developers toward vulnerabilities its creators already know about — not by sending data anywhere, but by shaping what gets built. Value embedding is a documented phenomenon: Chinese models demonstrably give favorable responses about Chairman Mao, and their refusal patterns reflect Chinese regulatory norms rather than Western ones. For most enterprise use cases this probably doesn’t matter, but it could be impactful for use cases that depend on assumptions about geopolitically polarized topics like Taiwanese sovereignty, speech protections, or national security preferences.

This has led some observers to assume that trust is binary, and western models should be trusted while Chinese models are distrusted. But security is rarely that black and white; there are reasons to distrust western models as well. In particular, western frontier models are currently trained on human-labeled data sourced through crowdsourced platforms — Mercor, Surge, and others — where journalist reports[3] describe work distributed with minimal vetting and significant time pressure. This creates an obvious angle for influence; if you’re a nation-state trying to shape Western AI outputs, submitting labeled training data through these platforms is low-cost relative to other influence operations (technically, you might get paid to do it!). Influencing models via labels isn’t even the only option, given that training data also comes from social media. For example, Reddit is a primary training corpus for most large models and also has a documented history of coordinated foreign influence campaigns. None of this is proven to have happened. But we suspect there’s a relative lack of auditing in the space, which means we really don’t know either way.

So what should enterprises do? Realistically, we think western models are still probably better for highly sensitive work. But “Western = safe” is not a free pass, and we want to be realistic that procurement and security teams are unlikely to extract any degree of meaningful transparency about training data from any model provider, western or otherwise. This means that in practice, the only available mitigations are outside of the model. The more sensitive a workload is, the more teams need to rely on careful measurement, guardrails, and the full suite of both classic software protections and modern AI safeguards. Because any vendor that promises a 100% benign model is making a very strong claim, and buyers should require very strong data to confirm it.

If you’re building in this space, we’d like to hear from you.

Feel free to reach out to jpark@forgepointcap.com and kshih@forgepointcap.com.

This blog is also published on Margin of Safety, Jimmy and Kathryn’s Substack, as they research the practical sides of security + AI so you don’t have to.