Earning the Right to Act: Putting Autonomous Agents into Production Operations

A support ticket lands for one of our customer-facing products. A customer’s record is wrong; something didn’t save the way it should have. Before a human even opens it, an AI agent has already read the product data, traced the inconsistency, and drafted the exact fix. It posts the proposed change into a Slack thread with two buttons: Approve and Reject. A support engineer reads the diagnosis, taps Approve, and the change goes live, after the backend re-reads the current record to make sure nothing has changed underneath it.

That loop runs in production today. So does a fulfillment agent that reads a stuck order, pinpoints the exact step it’s wedged on, and posts a root-cause summary directly onto the ticket. And so does the hardest one of all: an autonomous incident-response agent that investigates production incidents, builds a confidence-scored diagnosis, and can open its own draft pull requests. It has already surfaced a real bug.

None of this works because we trusted the models. It works because we spent most of our effort making sure they had to earn the right to act. What follows is a set of episodes from running these agents in real support and incident-response workflows, and what each one taught us about keeping a human on the button.

The button, and the read that happens after you press it

Generating an answer is the easy part. Modern models will happily produce a plausible-looking fix for almost any symptom you hand them. The hard problem is generating a safe, grounded diagnosis in a messy production environment: many repos, many services, and a web of hidden dependencies that no single prompt can see.

The first thing we got right was refusing to give the model a way to write. The moment you expose a raw write tool, “update this customer record,” to an LLM, you’ve handed it the keys. No matter how good your prompt is, a model with a direct mutation tool can bypass every approval gate you intended and change production state on its own. Prompt-level guardrails are not a security boundary. So raw write tools are filtered out of the LLM-visible toolset entirely. The model never sees a patch record function. It only sees a proposal tool, and it is structurally incapable of mutating production data because the capability isn’t in its hands to begin with.

# The model's visible toolset is filtered before it ever runs
all_tools         = [read_record, search_kb, propose_change, patch_record, ...]
llm_visible_tools = [t for t in all_tools if t.safe_for_model]
# patch_record is NOT in llm_visible_tools.
# The only path to a write is propose_change -> Slack approval -> backend applies.

That single design decision is almost embarrassingly simple, and it’s the most important one in the whole system. When the agent proposes a change, it lands in a Slack thread as an interactive message, governed by an explicit contract rather than a vibe. Approval only counts inside the thread where the proposal was made. Only members of an authorized support or updater group can approve. In the stronger version of the flow, the approver must be a different person than the proposer. And before applying an approved change, the backend does a fresh read of current state and compares it against the values the proposal expected. If the world moved on, the patch is rejected rather than allowed to clobber newer data.

That last piece matters more than it sounds. An approval can sit in a thread for minutes, and the record can change in that window. Re-reading before writing turns “approved” into “approved and still safe to apply.” The friction is real, and we accept it on purpose. Every mutation pays the latency of a human round-trip; for high-volume, low-risk changes that’s overhead, but for production data we decided it was the right default. The two-human rule bites on a small on-call rotation, so we treat it as a tunable property of the flow rather than a universal law.

The incident agent that argued with itself

The most humbling episode came from the incident-response agent’s own bug. On one incident, restarts and re-triggers each kicked off a fresh investigation with zero awareness of prior runs. The result was several divergent resolution posts and duplicate pull requests for a single incident: the agent arguing with itself in public.

The fix was unglamorous and exactly what you’d do for any production service. Scan for prior work before starting. Search for existing PRs tied to the incident and inject them into the prompt. Add an explicit deduplication instruction. Put an in-process guard around concurrent runs. The lesson stuck harder than the bug. An agent is production software. It deserves the same idempotency, state-awareness, and concurrency discipline as any other long-running service. “It’s just an LLM call” is how you ship a system that opens four PRs for one outage.

That episode also explains a deliberate scoping call. The incident agent runs read-only by default in its fully autonomous modes, and the write/approval flow is excluded from the incident path entirely, because that path can’t reliably surface a Slack approval at the moment it would need one. Different risk profiles, different contracts.

From a one-shot guess to evidence first

The incident agent didn’t start out careful. Its first version was a one-shot classifier that guessed a category from symptoms alone, with no confidence, no severity estimate, and no way to know what it didn’t know. We replaced it with a staged pipeline that refuses to classify until it has gathered evidence.

• Stage 1, Signal Intake. Fast pattern match on the raw alert. No API calls yet.
• Stage 2, Evidence Collection. Parallel telemetry queries: traces, logs, monitor definitions and history, upstream and downstream health. The instruction to the agent is blunt: collect ALL evidence before classifying.
• Stage 3, Classification. Multi-label, because a real incident can be partly a code bug and partly infrastructure. Each label carries a numeric confidence on a 0.0 to 1.0 scale, plus a separately-scored severity estimate (SEV1 to SEV4) with its own reasoning, a recommended action, and an explicit list of unknowns.
• Stage 4, Re-evaluation. As new evidence or engineer context arrives in the thread, the agent re-runs its assessment.

The subtle part is that confidence controls how the agent speaks, not whether it acts. High confidence posts a confident fix. Mid confidence posts a draft PR explicitly labeled “speculative fix, take a look.” Lower confidence posts findings, the unknowns, and suggested next investigation steps. The agent is useful at every certainty level instead of staying silent until it’s sure, and it never disguises a guess as a conclusion.

This framing only helps if people calibrate to it. A label that says “speculative” is worthless if engineers learn to rubber-stamp it, so we lean on the explicit unknowns list to keep the speculative path honestly speculative.

Why a smart model still wrote a wrong fix

The bug the incident agent surfaced is the part we’re proudest of, but the failures along the way taught us more. The worst was the agent that only sees the repo it’s in. Drop a coding agent into a single repository, hand it a stack trace, and it will confidently propose a fix that is locally reasonable and architecturally wrong, because it patched a symptom in the service it could see instead of the upstream dependency that actually broke. It coded before it understood the system.

Every time an agent produced a wrong fix, the root cause was the same: missing context, not a weak model. So we ground the agent in runtime telemetry, correlate recent deploys and merged PRs to connect an incident to the change that likely introduced it, and require it to consult organizational knowledge before proposing code. Behind that sits a living knowledge graph, auto-refreshed from every merged PR, plus repo-level guidance files that document each service’s external dependencies and conventions. The agent reads how the system actually fits together before it writes a line. Feeding it telemetry, deploy history, and a knowledge graph moved the needle far more than any prompt tuning ever did.

What the episodes add up to

The payoff has been concrete. AI review now runs early on changes rather than as an afterthought, and the path from review to production is dramatically shorter than it used to be. Test runs that once took many minutes now complete in a fraction of the time. Monitoring checks increasingly generate themselves, and prototypes move from idea to working software far faster than the old hand-built cycle.

The durable result isn’t the speed. It’s the pattern. Autonomy and safety aren’t opposites you trade off against each other; they’re two things you engineer together. Constrain what the model can touch. Make it gather evidence before it concludes. Let confidence govern how loudly it speaks. Keep a human on the button for anything that mutates the world. Do that, and an agent doesn’t have to be trusted blindly to be genuinely useful. It earns the right to act, one grounded step at a time.