Why Agent Trust Belongs in the Operating System
Why Agent Trust Belongs in the Operating System
Every few weeks a new AI coding tool ships a feature meant to make its agent safer to run: an approval prompt before it executes a command, a sandbox mode, a log you can scroll if you know where to look. Each is a reasonable patch. Together they add up to a strange picture: we are trying to make agents trustworthy one application at a time, and it isn’t working.
It isn’t working because trust is not a feature you can add to an app. It’s a property of the system the app runs in. And right now, the system — the operating system — was designed for a completely different job.
The OS we have was built to help a human drive apps
Take stock of where a modern desktop actually spends its budget. A compositor. A window manager. An application model. A constellation of background daemons. Almost all of it exists to help a person drive a fleet of applications with a mouse and a keyboard. That was the right design for forty years. The human was the intelligence in the loop; the machine’s job was to render windows and get out of the way.
An AI agent inverts that arrangement. The agent is now the thing doing the driving. But it’s driving through an interface built on the assumption that a careful human is at the wheel — one who won’t rm -rf the wrong directory, won’t paste a secret into the wrong box, won’t be talked into something by a malicious web page. The agent has none of those instincts, and the OS offers it none of those guardrails, because the OS was never asked to.
So every app that ships an agent has to reinvent the safety layer itself. One tool builds an approval prompt. Another builds a sandbox. A third pipes everything through a container. They don’t share a model, they don’t share a log, and none of them can constrain what happens outside their own process. The moment the agent reaches for a shell, a file, or the network, it’s back in an environment with no memory of the app’s good intentions.
Trust is a systems property
Here’s the part that’s easy to miss when you’re focused on shipping the next feature: the properties we actually want from an agent are not app-level properties. They’re system-level.
Consider what “I trust this agent on my machine” really decomposes into:
- Least privilege. The agent can do exactly what a task requires and nothing more. That’s not something an app can grant itself — privilege is defined by the system that hands out access to files, devices, and the network.
- Accountability. Every consequential action is logged, timestamped, attributable, and replayable. A per-app log is a diary the app writes about itself. Real accountability has to live below the app, where it can see every action regardless of which tool initiated it.
- Containment. A compromised or confused agent can’t quietly escalate. Containment is a boundary, and boundaries are drawn by the OS, not by the process asking to be contained.
- Durability. The record of what happened survives a crash, a reboot, a closed tab. State that lives inside one app dies with that app.
Every one of these is a statement about the environment the agent runs in. You cannot patch them in application by application any more than you can make a city safe by asking each driver to bring their own traffic lights.
What “capability-based” means, and why it’s old
The good news is that we are not inventing the answer from scratch. The idea that a program should hold unforgeable tokens of authority — capabilities — rather than acting with the ambient power of whoever launched it goes back to the 1960s. Capability-based security is one of the most durable ideas in systems design: instead of “you are root, so you can do anything,” it’s “you hold a token that lets you write to this directory, and nothing else.”
For decades that model was mostly academic, because for a human-driven desktop it was overkill — the human was the policy. Agents change the economics completely. When the thing driving your machine is a probabilistic model that can be wrong or manipulated, “acts with the full authority of the user who launched it” stops being convenient and starts being the whole problem. A default-deny, capability-brokered system — where every consequential action has to present a token for exactly the authority it needs — is suddenly the natural fit. The forty-year-old idea was waiting for this workload.
Local-first isn’t nostalgia — it’s part of trust
There’s a second systems property that matters here, and it’s easy to dismiss as a preference: where the work runs. A trustworthy agent should be able to do its job local-first — against a model on your own machine — with the big hosted models on tap when you want them, not sitting on the critical path.
This isn’t nostalgia for offline software. It’s continuous with everything above. If your agent can only function by shipping your files and your context to someone else’s server, then “least privilege,” “containment,” and “accountability” all have an asterisk: except for everything we sent away. Keeping the default local keeps the trust boundary somewhere you can actually see. That argument gets its own treatment in keeping local-first AI off the critical path.
The uncomfortable conclusion
If you accept that least privilege, accountability, containment, durability, and a local default are the real ingredients of agent trust — and that all five are properties of the system rather than the app — then the conclusion is uncomfortable but clean: the right place to solve agent trust is the operating system. Not a wrapper around one editor. Not a sandbox bolted onto one tool. The layer that hands out authority, draws the boundaries, and keeps the record.
That’s the bet behind an agent-native OS: stop treating the agent as an application that needs guardrails, and start treating it as the thing the system is built around — with the authority model, the log, and the boundaries designed for it from the floor up. Delivery windows instead of application windows. A typed broker instead of a raw shell. A record that survives the reboot.
We’re building that in the open, in Rust, under Apache 2.0, and we’re honest about the phase: the alpha replaces your desktop session, not your kernel, and there’s a long road to the full picture. But the direction follows from the argument. You can keep patching trust into apps one prompt at a time. Or you can put it where it actually lives.