Measuring Human Intervention: The Falling Rate Earns the Next Gate Reduction

Intercom auto-approves over 19% of its pull requests with no human reviewer and recorded zero reverts across a 100-plus PR pilot, which is the whole case for measuring human intervention: a system that asks for less steering and still ships clean is a system that has earned the next gate reduction. Outcomes show the work is good. The intervention rate shows how much human steering it still takes to get there, trending over time. Both numbers come out of the same place, the audit trail the environment already keeps, so neither is a metric you go build a pipeline to collect.

The metric is interventions per unit of work, tracked over time

Define it flat. The intervention rate is the fraction of agent-produced units of work that required a human to step in: a human review requested, an auto-approval declined, a takeover, a revert, a correction pushed on top. Pick the unit that matches your gate. Intercom measures it per pull request, because the PR is where its decision gate sits and where the agent either clears the bar or gets routed to a person.

The number that matters is the trend, not the level on any single day. Intercom's published split is the worked example: over 19% of PRs are auto-approved with no human reviewer, and that share ramped over the first four weeks of broader rollout. The 19% is the share of work the system no longer steers by hand, and the four-week ramp is the trend. The other 81% still stops for a person, which is the point of tracking the rate: it tells you how wide the no-human-needed band has grown, not that the gate is gone.

The falling rate is the evidence that widens auto-approval

A falling intervention rate is not a vanity metric. It is the input to a specific decision, management by exception: define a tolerance band, let anything inside it proceed automatically, and spend human attention only on what falls outside. The intervention record, trending down for a given class of change, is the evidence that the band can widen to include that class. You do not lower a gate on a hunch. You lower it because the recorded rate of human corrections on that class of work has held low long enough to justify the next reduction.

Intercom's pilot is the proof shape an auditor will accept: over 100 PRs through the approval pipeline, zero reverts of AI-approved PRs, and a 6-16x improvement in time-to-approval at the 75th percentile. The zero-reverts figure is the safety side of the same coin as the 19% auto-approval rate. Auto-approval went up and reverts did not, which is exactly the pattern that earns more auto-approval. This is the feedback loop that auto-approval by risk runs on: risk classification decides which gate applies, and the measured intervention rate decides when a class graduates to a wider exemption.

The arithmetic also explains why the metric exists at all. Review capacity is the bottleneck that moved past the merge: generation got cheap, human verification did not, and when throughput outruns capacity, review hollows out into rubber-stamping. The intervention rate is how you confirm you are reallocating human attention rather than quietly abandoning it. A band that widened because the evidence supports it looks identical, on the surface, to a band that widened because the queue backed up. The number is what tells the two apart.

The metric is exhaust from the audit trail, not a new pipeline

You do not instrument the intervention rate. It is already in the record. Every review decision, auto-approval, takeover, and revert is an event in the session log, the same append-only trail that lets a crashed agent resume and that an auditor samples for evidence as a byproduct. The intervention rate is that trail filtered to the human-action events and divided by the total. It resolves along the same axes as every other audit query, by task, by actor, by environment, by time window, so the four-axis query that answers "show me the trail for this decision" also answers "what fraction of this quarter's agent PRs needed a human." If your record cannot answer the second question, it cannot fully answer the first either, and that gap is worth fixing before you debate the rate.

This is the same argument Measuring Outcomes makes for revert rate and time-to-approval, pointed at steering instead of quality. Both are leading candidates for the metric of record precisely because the environment produces them as a byproduct of normal operation.

The rate is gameable, and low is not the goal if quality drops

Two caveats keep the metric honest.

First, the intervention rate is gameable in the direction everyone wants it to go. The fastest way to drive interventions to zero is to stop reviewing, which is the rubber-stamping failure mode review capacity documents: the SOC 2 control still appears in the binder, the box still gets checked, and nothing is actually being controlled. An intervention rate read in isolation cannot tell a healthy low number from a hollowed-out one. METR's randomized trial is the general warning against trusting a self-flattering number: developers overestimated their AI speedup by over 40 percentage points while a stopwatch measured them slower. A falling intervention rate is just as capable of measuring your confidence rather than your safety.

Second, low intervention is not the objective. Clean shipping is. The intervention rate only means something paired with a quality signal that cannot be gamed by reviewing less: revert rate, change-failure rate, delivery stability. Intercom pairs its 19% auto-approval with the production comparison that backs it, AI-authored backend code reverting at 0.53% versus 5.39% for human-authored, frontend at 0.22% versus 2.00%. The 2025 DORA report supplies the population-level reason to insist on the pairing: across nearly 5,000 respondents, AI adoption correlates positively with throughput but still negatively with delivery stability. Volume can rise and stability can fall at the same time. An intervention rate dropping while reverts climb is not progress toward autonomy. It is a queue backing up with the alarm disconnected.

The short version