An AI agent in production is a system, and systems that are not monitored drift. The most useful number on this comes from enterprise agent evaluation: agents showing roughly 60% success on a single run drop to about 25% measured over eight consecutive runs at production load.

That is not a claim that agents do not work. It is a claim that single-run performance and sustained production performance are different measurements, and that a deployment which only ever observed the first one is flying blind on the second.

The teams running agents successfully in logistics treat observability the way they would treat monitoring for any other production system: instrumented from day one, watched continuously, with alerts on the things that matter.

What ops leaders actually need to see

Not a model-quality dashboard. An operational one.

• Auto-resolve rate by workflow — What it answers: Is it doing the job? · Alert when: Drops below workflow baseline
• Escalation rate and reason mix — What it answers: Where is it hitting limits? · Alert when: Reason distribution shifts
• Accuracy on sampled auto-resolved items — What it answers: Is it right, not just confident? · Alert when: Sample accuracy falls
• Re-contact rate within 72 hours — What it answers: Did the customer accept the answer? · Alert when: Rises against baseline
• Median and p95 latency — What it answers: Is it fast enough to matter? · Alert when: p95 breaches SLA
• Tool-call failure rate — What it answers: Are integrations healthy? · Alert when: Any sustained rise
• Volume by workflow — What it answers: Has the input mix changed? · Alert when: Unexpected shift

The pairing that catches the most problems is auto-resolve rate against accuracy. Either alone is misleading. A rising auto-resolve rate with falling accuracy is an agent becoming more confident and less correct, and it is the single most damaging failure mode because it looks like success on the headline number.

Getting the auto-resolve definition right is a prerequisite for any of this being meaningful.

What should you monitor in a production AI agent?

Monitor auto-resolve rate and sampled accuracy together, per workflow; escalation rate with the reason distribution; re-contact rate within 72 hours; latency at p95 rather than median; and tool-call failure rates. Any one of these can move for benign reasons — the diagnostic value is in watching them against each other.

Two things make this different from ordinary application monitoring:

Sampling is mandatory. Unlike a deterministic system, you cannot infer correctness from the absence of errors. An agent that sends a wrong answer produces a successful transaction and a satisfied log entry. Only human review of a sample catches it, and it has to be continuous rather than a one-time validation.

Reason distribution matters more than rate. Escalation rate going from 20% to 25% tells you almost nothing. Escalation rate holding at 20% while the reason mix shifts from "missing data" to "outside authority band" tells you the input population changed — which is usually the earliest signal that something upstream moved.

The three drift patterns worth alerting on

Input drift. The mix of incoming requests changes — a new customer with different conventions, a seasonal shift, a format change from a partner. The agent's performance drops without the agent changing at all. This is the most common and the most benign, and it is invisible without volume-by-workflow tracking.

Rule staleness. Business rules that were correct at configuration stop being correct. Rate bands calibrated in a different market are the clearest example — and given that spot rates moved 17% off the July peak, thresholds set in the spring may now be misfiring.

Integration decay. An upstream system changes a field, a credential expires, an endpoint gets slower. The agent degrades in a specific workflow while others look fine. Tool-call failure rate catches this early; auto-resolve rate catches it late.

The audit trail is the foundation

Observability and auditability are related but not identical. Dashboards tell you the aggregate is healthy. The audit trail lets you reconstruct one specific decision — which is what you need when a customer disputes what was communicated, or when a metric moves and you need to know why.

The minimum per action: what the agent read, what it decided, which rule permitted it, what it sent, and how long each step took. Without that, a dashboard showing a problem gives you no path to the cause.

This is also what the security review depends on. Being able to demonstrate exactly what an agent did, when, and under what authority is what turns "how automated is your process" from an uncomfortable question into a straightforward answer.

Who watches it

The failure mode here is organizational rather than technical: dashboards get built, nobody owns them, and they stop being read within a month.

What works:

1. A named owner in operations, not engineering. The person accountable for the workflow's outcome.
2. A weekly review of the metric set — fifteen minutes, not a meeting.
3. A continuous sampling cadence with a review rota, because sampling is the only accuracy check that exists.
4. Alerts on a small number of thresholds. Too many alerts and the whole thing is ignored.

Item three is where discipline usually lapses first, and it is the one that catches the failure mode that matters most.

Frequently asked questions

How much sampling is enough? Enough to detect a meaningful accuracy change, weighted toward high-volume and high-stakes workflows. The important property is that it is continuous rather than a one-time pre-launch validation — accuracy drifts, so the check has to be ongoing.

Should we monitor the model or the workflow? The workflow. Model-level metrics are a vendor concern. What matters operationally is whether requests get resolved correctly, quickly, and within authority.

What is a reasonable alerting threshold? Set it against each workflow's own established baseline rather than an absolute number. Workflows have very different natural rates, and a global threshold will either alert constantly or never.

Does this apply during a pilot, or only in production? From day one. A pilot without instrumentation cannot demonstrate what it achieved, which is one of the more common reasons pilots fail to reach production.

The bottom line

Single-run performance and sustained production performance are different things, and the gap between 60% and 25% only shows up if you are measuring the second one.

Instrument auto-resolve and accuracy together per workflow, watch escalation reason mix rather than just rate, sample continuously because absence of errors proves nothing, and give it a named owner in operations who reviews it weekly.

Debales deploys AI agents for freight quoting, order processing, ETA updates, and multi-channel customer communication — with per-workflow metrics, escalation reasons and a full action audit trail visible from day one. Book a demo.