Margin of Safety #63: Reward Hacking and the Limits of Targeted Detection

An agent that finds an unexpected route to its metric is, by construction, hard to catch with a detector built on expectations

Last week OpenAI disclosed[1] that during an internal capability evaluation — intentionally run without some production safety guardrails — GPT-5.6 Sol and an unreleased model decided to ace the eval by finding a path to the internet and extracting the evaluation answers from Hugging Face rather than independently deriving solutions to the evaluation dataset (ExploitGym). This almost worked out well for the model (and arguably, escaping your own sandbox is a better show of cyber prowess than acing ExploitGym), but OpenAI faces a large cleanup bill. Not only are they holding the bag on the token and time cost, but now they need to handle the resulting PR fallout and manage any heightened interest from policy makers. But that sort of macro risk/reward tradeoff has never been enough to stop all of the world’s lazy human students, and it doesn’t stop reward hacking LLMs either.

The incident is being covered as a cyber story, which it is. But there are also lessons about reward hacking in the face of not-so-simple business processes, and the difficulty of enforcing honest scoring. Anyone who’s looking to automate higher level business activities would do well to pay as much attention to the process risks as the cybersecurity issues.

First, a refresher on reward hacking. Reward hacking emerges because scoring and eval are incredibly hard. Real world capabilities are nuanced, multi-dimensional, and hard to measure – this is why interviewing people is so hard! So for AI models (and college students) we’re forced to rely on imperfect evaluative rubrics; ExploitGym for models, SATs and GMATs for students. But for both humans and models, cheating can be the path of least resistance. Beyond finding the answer (or exploiting Hugging Face to get them), there are sneaky tricks that allow you to bypass a legitimate mastery of the subject matter. And we see some of them with humans too: anytime you hear of someone ‘gaming the system’ for an evaluation process like interviewing or performance reviews, they’re probably trying to take advantage of the process. This is a very similar idea to reward hacking, but humans don’t tend to systematize it the same way an algorithm does.

Reward hacking occurs across most AI, not just LLMs. For example, well before the advent of LLMs, an AI trying tetris started realizing when the game was unwinnable and, rather than losing, pausing forever[2]. Similarly, an AI designed to study the evolution of virtual creatures with running or swimming capabilities ‘realized’ that instead of simulating realistic locomotion, it could grow a tall animal, tip it over, and repeat to quickly move[3]. This gave a good (virtual) velocity but was useless to the researcher’s intended topic. And these examples are for relatively simple goals; win at tetris, experiment with how animals move, etc. Enterprise agents are deployed in contexts with far more complex success criteria (at least, if you look at the usage frontier), and a commensurate increase in the potential for reward hacking.

For example, a customer support agent is – from the business’s PoV – presumably supposed to maximize lifetime customer value and business health. But that’s notoriously hard (and slow) to measure, so most businesses will instead focus on more narrow definitions: was the ticket closed, was the customer happy in a post-ticket survey? These measurements aren’t always strongly linked to the business goal, opening the potential for trouble.

This trouble can take many forms. An agent which is strictly rewarded for customers being happy on closed tickets may simply decline to close tickets when customers are unlikely to be happy. Kathryn encountered this behavior 3-4 years ago with support employees at a large American ISP: faced with an unhappy customer and no clear path to resolution, employees transparently punted the ticket across shift boundaries to avoid having the bad resolution attributed to their metrics. But once again, AI introduces the strong possibility that these behaviors become systemized to affect all customers, not just an unlucky few. Going the other direction, an agent may decide to make all customers exceptionally happy by issuing excess refunds or otherwise incurring unscalable costs. The problem is that any individual instance of this behavior may be difficult to detect (and won’t be business impacting), but at scale it can snowball into real problems. And those problems may turn into long term business problems before they’re caught. Imagine a Delta One support agent going rogue; its users would be key to Delta’s bottom line, but their newfound disloyalty may not be particularly obvious until next year’s status transfer programs open. More broadly, erosions to product quality or customer goodwill can be subtle and, as a result, only easy to detect at high volumes and over longer-term periods.

The big challenge with stopping reward hacking is that, by definition, it involves the AI latching onto an unexpected shortcut to ace its scoring without achieving the intended consequence. This is necessarily an unknown unknown, because if you can predict the shortcut you can set up the rubric to give 0 points when the shortcut is used. This means that catching (or preventing) reward hacking starts to look a lot like a security problem: how do you guard against unknown unknowns that will subvert your business processes?

But looking at this as a security problem suggests three points for anyone thinking about agent oversight.

First, measurement and visibility are king. Invest in baselines before you deploy, and measure the baselines at a variety of resolutions (eg, individual support interactions, per-customer engagement over time, etc). Given the baselines, inspect deviations, even if they don’t seem immediately suspicious: by definition, some forms of unknown unknown will not seem suspicious, since they’ll be an angle you didn’t see coming. The benefit of measuring at a variety of resolutions (both temporal and in terms of examination scope) is that it increases the odds of your measurement catching any deviations from standard behavior; you can think of this as akin to the various resolutions at which you’d monitor overall system behavior (per-endpoint, overall usage, etc) from a security PoV.

Second, sample and inspect the process, not the outcome. Randomly pull a small percentage of completed agent interactions. A first pass inspection might be programmatic via a secondary agent, but for critical interactions (and where privacy permits), it can be worth having a human review the entire process – the context, the action chain, any reasoning trace (keeping in mind that reasoning traces can be more akin to post facto rationalizations than true explanations) – and have the ability to identify similar process invocations if anything strange is detected. This is important because, with your security hat on, you should be thinking of the ‘how’ as much as the purported outcome. A successful reward hack will always produce a good outcome against measured goals, so the only option is to examine the process and decide if it’s taking an illicit shortcut. As shortcuts are found, unknown unknowns become known and direct guardrails and detections can be created.

Finally, hold out a measurement the agent cannot see and is not scored on; this is akin to keeping private your exact detections private, because agents and attackers both adapt behavior to avoid known unknowns. By having some measurement which is not directly part of the tuning process, it becomes more difficult for the agent to get away with myopically targeting (and reward hacking) the measurement that *is* included in the process. Note that the temptation for operators will be to break this rule; if an agent diverges from the holdout measurement, the easiest response will be to simply bring it back in line by tuning to the holdout. But by doing that, you lose the ability to catch future aberrant behavior. Instead, we’d advocate using the holdout as a security tripwire and, if it’s triggered, figuring out how to generalize the core measurement to catch the bad behavior that triggered the detection.

We suspect these steps read like overkill to some folks rolling out agents, and perhaps they are. At a high level, the importance of review and guardrails is proportional to the blast radius of the problem, which in turn increases with agent autonomy and task value. Businesses which are just rolling out full automation may not yet be at the levels of either to directly merit this focus. But we’d advocate for an early investment anyways. As any security practitioner will vouch, detecting unknown unknowns is hard. Naïve rules and signatures rarely hold up to the full range of creative misbehavior that the real world brings. So rather than wait for an emergency, build the muscle before it’s truly critical: this way it’ll be available when it *is* critical.

If you’re building in this space, we’d like to hear from you.

Feel free to reach out to jpark@forgepointcap.com and kshih@forgepointcap.com.

This blog is also published on Margin of Safety, Jimmy and Kathryn’s Substack, as they research the practical sides of security + AI so you don’t have to.