Measure the Outcomes the Pipeline Already Logs, Not the Vibes

In METR's 2025 randomized controlled trial, experienced open-source developers using early-2025 AI tools took 19% longer to finish their tasks while estimating the AI had sped them up, a gap between perception and stopwatch of over 40 percentage points. The compliance case earlier in this section rests on the environment emitting an audit trail by default; the measurement case rests on the same audit trail. Track the outcomes the pipeline already produces as evidence, revert rate, time-to-approval, delivery stability, and cost per change, because those numbers are recorded whether or not anyone feels productive.

Perception runs 40 points ahead of the stopwatch

The original RCT measured a 19% slowdown (95% CI: +2% to +39%) while participants perceived roughly a 32% time uplift. METR's May 2026 follow-up survey of 349 technical workers found the same pattern at larger magnitudes: a median self-reported 3x speed gain and 1.4 to 2x value gain, with respondents forecasting 2.5x by March 2027. METR's own checks undercut the high end. Of 7 respondents claiming 10x or more on at least two value measures, the 2 with viewable public outputs had, in METR's judgment, confidently overstated their gains. METR staff, who know the perception-gap finding, gave the lowest estimates of any subgroup. Source: METR.

Self-report is not noise around the true value; it is biased upward, consistently. The median survey respondent claimed they would give up 29% of their salary to keep AI access for a month. People who feel that strongly are not neutral instruments.

The corrected number is "we don't know"

The 19% slowdown is also out of date, and the honest replacement is wide error bars. METR's February 2026 continuation (57 developers, 143 repos, 800+ tasks) found directional speedup: −18% (CI −38% to +9%) for returning developers, −4% (CI −15% to +9%) for newly recruited ones. METR then declared its own estimate unreliable. Developers increasingly refused to work without AI at all, and 30% to 50% said they withheld tasks they did not want to do unassisted, systematically removing the highest-uplift work from the sample. METR calls the estimate a likely lower bound and is redesigning the experiment. Source: METR.

Task-level speedup is genuinely hard to measure even for a dedicated research lab, so your internal dashboard of vibes-adjacent numbers is not going to do better. Measure downstream outcomes instead.

Revert rates and time-to-approval are measurable today

Two organizations have published outcome numbers worth copying.

Intercom instrumented its AI approval pipeline and published the sharpest outcome metric in the corpus: AI-authored backend code reverts at 0.53% versus 5.39% for human-authored, roughly a tenth the rate, with time-to-approval and full revert and auto-approval splits in the case study.

The 2025 DORA report (nearly 5,000 respondents) supplies the population-level caution: AI adoption now correlates positively with delivery throughput and product performance, but still negatively with delivery stability. More than 80% of respondents believe AI increased their productivity; 30% report little or no trust in its code. Both can be true, which is why you track stability, not belief. DORA's framing holds up: AI amplifies what is already there, and the gains land on teams with strong feedback loops and platforms.

Revert rate, time-to-approval, and change stability are the metrics to trust because the environment produces them as a byproduct: every merge, review decision, and rollback is already in the audit trail. This is evidence as a byproduct pointed at metrics instead of auditors, and it runs over the same four-axis query the section uses to answer who, what, when, and on what. If your pipeline cannot answer "what fraction of AI-authored changes were reverted last quarter," fix that before debating productivity multipliers. See verifiable outcomes for the upstream half: making each task measurable in the first place.

Cost per merged change is the metric nobody publishes yet

The field is thin here, and most of what exists is forecast rather than measurement. Gartner's May 2026 Magic Quadrant for Enterprise AI Coding Agents reports, from its 2026 Software Engineering Content Survey (482 respondents), that 90% of engineering leaders see productivity gains attributable to AI, with a net average gain of 19.3%. Gartner sizes the market at $9.8 to $11.0 billion annualized as of April 2026 and predicts that by 2028, AI coding costs will overtake the average developer's salary as consumption-based pricing and token usage rise. That prediction is a planning assumption, not a measurement; treat it as a scenario to budget against.

Gartner's measurement guidance matches METR's lesson from the other direction: baseline against DORA and SPACE, and avoid activity and volume metrics (lines of code, acceptance rates) that "often fail to capture broader, system-level productivity gains." Their summary of 2026: the ROI conversation is "less about whether enterprise AI coding agents deliver value, and more about at what cost that value is realized." Nobody in the corpus publishes cost-per-merged-change; the field is silent on it. For the published figures that do exist, see honest numbers.

Reject the vibes, use the four numbers the pipeline emits