Margin of Safety #59: The Level 3 Trap… Why AI Agents Are Burning Out Knowledge Workers

Current AI agents cause burnout by trapping workers in an unsustainable “Level 3” monitoring state — retaining accountability while losing control

Over the past few months, we’ve noticed a shift in messaging from the foundation labs. Compare Dario Amodei’s October 2024 prediction that AI could eliminate “50% of entry-level white-collar jobs within one to five years” to Daniela Amodei’s February 2026 ABC News interview, where she argued that the number of jobs AI can handle without human involvement is “vanishingly small” and that “humans plus AI together actually create more meaningful work.” The underlying technology hasn’t changed direction — but the public framing has.

This may be right. But we’ve both been experimenting with AI, and think we need to rethink how we design agent harnesses and human/agent interactions if we’re going to achieve that.

The core problem is one of sustainability. Steve Yegge — veteran opinionated engineer — has been sounding the alarm on AI-driven burnout since his February 2026 essay “The AI Vampire.” His observation: even engineers running at peak AI-augmented output hit a wall at roughly three hours of productive work per day, after which the cognitive cost of directing, reviewing, and correcting agent output overwhelms any gains. This isn’t unique to software. A February 2026 UC Berkeley/Yale embedded study at a 200-person tech firm found that AI adoption was intensifying work rather than reducing it — more information to process, weaker boundaries between work and non-work hours. A 2024 Upwork survey of 2,500 knowledge workers across the US, UK, Canada, and Australia found 77% felt AI tools had actually decreased their productivity and added to their workload, against 96% of their executives who expected the opposite. We feel it too. Agents are powerful, but they don’t currently fit into a mode of productivity that matches most human brains.

Long-term, we think that’ll be a problem. For a tool that supports human workers to be valuable, it must improve total productivity. But if a tool contributes to burnout, the math gets complicated fast. Churn is expensive: even if hiring is easy (which we dispute in an era where deepfake candidates can pass early screening rounds), you still bear the cost of interviews, onboarding, and the ramp before a new employee reaches full productivity. At the end of the day, the tooling that uplifts versus fries your team should win.

We think there are a few reasons these stresses are emerging.

The first is that agents are improving rapidly at hard skills — things we can score and measure, which makes them tractable for reinforcement learning, labeling, and other standard model improvement techniques — but not soft skills. When we interact with teammates in a functional way, soft skills do a lot of load-bearing work by default. We calibrate: something urgent enough can interrupt a colleague’s dinner; something that can wait does wait. We think much of the friction in current agent interactions comes from the fact that agents make human-style demands without human-style social affordances. They’re not good at knowing what they don’t know, and putting the onus on operators to slowly and explicitly declare all preferences in configuration is challenging; it’s like the intern who requires you to enumerate every social grace you expect them to possess.

To understand the second issue, we need a framework from a different industry. The Society of Automotive Engineers (SAE) defines six levels of vehicle automation, from Level 0 (fully manual — the driver does everything) to Level 5 (fully autonomous in all conditions, no human required). Levels 1 and 2 are driver assistance: the car helps with steering or speed, but the human is actively in control and watching the road. Level 3 is where the car can manage the driving task within defined conditions — highway driving, slow traffic — but the human must stay ready to take over when prompted. Level 4 removes the intervention requirement within a specific operational domain (a geofenced city area, for instance). Level 5 handles everything, everywhere, with no human in the loop at all.

Level 3 has turned out to be a trap. In 2013, Google (pre-Waymo) ran tests with trained engineers who were explicitly told to stay alert and take over when prompted. It found that they eventually climbed into the back seat, fell asleep, and watched videos while traveling at highway speeds. They were unable to modify the experience to prevent this, even with all the alarms and tests in the world. Waymo CEO John Krafcik later called Level 3 “a myth”; they concluded that the human brain naturally disengages when monitoring mostly-correct AI and is systemically unable to maintain attention in order to supervise such systems. Waymo dropped Level 3 and jumped straight to Level 4. Ford followed the same reasoning for the same reasons.

Combine these factors, and we think you get to the issues Steve Yegge and others are flagging. A lot of things can drive burnout, but we have a hypothesis: Accountability – Control = Burnout. The challenge is that many firms are rolling out AI in contexts where employees retain accountability but can only retain control if they closely supervise what is effectively Level 3 automation while babysitting a brilliant but oblivious intern. This is a novel demand and there’s preliminary evidence that many people find it uniquely and perhaps unsustainably taxing.

The automotive industry’s experience with Level 3 offers a clear design principle: don’t build systems that require sustained human vigilance over processes with infrequent but serious failures, because you’re asking for the impossible.

This means Level 3 must be a transitional state (if you need it at all), not a product. Harnesses need to aim for Level 2, with sustainably interactive human engagement, or Level 4, where the agent operates autonomously within a well-defined domain and the human checks out.

The second implication is for social architecture. Current agents make human-style demands without human-style judgment about what warrants interruption. A useful test: what would a good intern do differently from a bad one? A bad intern produces volume and surfaces every question. A good intern develops calibration; they learn what you care about, front-load the decisions that need you, and handle the rest. And importantly, they don’t require you to explicitly tell them what they need to do differently every time they mess up. This isn’t something that foundation models are likely to fully subsume, both because it implies individual learning about all the humans on a team and also because it’s a task that isn’t (yet) amenable to bulk scoring and reinforcement learning.

The firms that figure out sustainable human-agent collaboration, not just peak quantity output, will have a durable advantage over those optimizing for throughput. Tooling that uplifts teams will win over tooling that burns them out, even if the latter looks more impressive in a demo.

If you’re building in this space, we’d like to hear from you.

Feel free to reach out to jpark@forgepointcap.com and kshih@forgepointcap.com.

This blog is also published on Margin of Safety, Jimmy and Kathryn’s Substack, as they research the practical sides of security + AI so you don’t have to.