Verifiable Outcomes or It Doesn't Ship

Airbnb converted 3.5K Enzyme test files to React Testing Library in six weeks, against a 1.5-engineer-year estimate, with 97% automated success. Stripe merges 1,300+ agent PRs per week. Spotify's Honk reached roughly 1,000 merged PRs every 10 days. Not one of these wins is open-ended feature work. Every one is a migration, an upgrade, or a maintenance task, and every one runs against a check that says pass or fail without asking a human. Survey the published record and the rule holds without exception: the autonomous successes are narrow tasks with hard validation loops. A task without a verifier is not autonomous work yet.

The companion page, closing the feedback loop, is about wiring the verdict back to the agent so it can read CI, tests, and review bots and fix itself. Does a machine-checkable verdict exist at all? It decides which work is safe to route to an agent in the first place, so it sits at the front of managing the work, ahead of the decision gates and the takeover rules that follow.

Every published win names its verifier

The task types repeat across companies: migrations, test coverage, dependency upgrades, config changes. Spotify's list is representative: language modernization, breaking-change upgrades, UI component migrations, YAML and JSON parameter updates. For these, Spotify reports 60–90% time savings versus writing the code by hand.

The vendor selling the most autonomy says the same thing. Cognition's 2025 Devin performance review reports hundreds of thousands of merged PRs at a 67% merge rate, frames the ideal assignment as "clear, upfront requirements and verifiable outcomes," and mandates human review because "code quality is not straightforwardly verifiable."

The verifier converts unpredictable output into shippable output

An agent's output is a guess; a verifier turns the guess into a fact. Spotify states the problem plainly in the Honk write-up: agent output "can be unpredictable," which "creates a need for new validation and quality control mechanisms." Their architecture is the answer in miniature. Honk replaced the deterministic transformation script with a prompt, but kept all the surrounding Fleet Management infrastructure, targeting, PR creation, review, and merge, "exactly the same." The generator became probabilistic. The gate did not.

The before-and-after explains why this trade works. Spotify's deterministic Maven dependency updater grew to over 20,000 lines of code handling corner cases, and only a few teams had the expertise to write transformations like it. The agent collapsed the authoring cost to a prompt, and correctness was never the script's job alone: the tests, the judge, and the review gate decide what merges. Cheap generation pairs with hard verification. Closing the feedback loop covers the plumbing that delivers the verifier's verdict back to the agent.

The analysts and the practitioners are both right

Thoughtworks holds software engineering agents at Trial, favoring supervised in-IDE use over autonomy, and Birgitta Böckeler's experiments document constant intervention as the price of quality. Meanwhile Stripe and Spotify merge thousands of unsupervised-by-default PRs a month. These positions look contradictory and are not. The reconciliation is scope: the analysts are testing open-ended work, and the practitioners shipped narrow work with verifiers. Böckeler's findings are the accurate report on what happens when the task has no hard check: a human becomes the verifier, intervention by intervention. Stripe's and Spotify's numbers are the accurate report on what happens when it does.

Ramp's Inspect sits deliberately between the two: ~30% of merged PRs with no mandate, and engineers watch and steer live from Slack, web, or a Chrome extension. The agent gets the same databases, CI, Sentry, and Datadog the engineers have. Where the verifier is partial, Ramp keeps a human in the loop and makes the loop cheap.

Scope to the verifier you have

The operational rule for work intake follows directly: before assigning a task to run autonomously, name its verifier. If the answer is a test suite, a compiler, a schema, or a coverage report, the task is a candidate. If the answer is "someone will look at it," it is supervised work, and the human is the verifier whether you planned for one or not. The third option is Airbnb's: build the verifier first. Their 97% number came from a state-machine pipeline with retry loops, not from a better prompt. Scoping work covers how to break ambiguous requests into pieces that have one.

One fence: the published record contains no autonomous success on ambiguous, greenfield product work. The field may get there; as of mid-2026 nobody has shown it, and the teams running the largest fleets, Spotify included, say out loud that "we don't have all the answers yet." What every shipping system has instead is a verifier, and then review as a gate behind it.