How to Measure the ROI of AI Coding Tools (Beyond Acceptance Rate)
The research on AI coding assistants is mixed, and most dashboards report activity instead of delivery. Here's a three-stage way to measure what the tools change, plus an ROI model finance can check.
Executive brief
- AI coding assistants are now near universal, with about 90% adoption in DORA's 2025 survey, but the published evidence shows that individual speedups don't automatically become faster, more stable delivery.[1][2] Measure the path from first commit to production and on into maintenance, not tool activity.
- Faros AI's telemetry on more than 10,000 developers found 98% more merged pull requests alongside 91% longer review times and no measurable improvement in organization-level delivery metrics.[8][9] Speeding up one step moves the queue to the next one.
- Stop reporting suggestion acceptance rate, lines of code added and time spent in the tool.
- Track three stages. Development (time to PR ready, PR size, 30-day churn, review load), deployment (lead time, mutation score, CI first-pass failure rate, security findings per PR) and day-2 maintenance (change failure rate, rework rate, recovery time, migration speed).
- Treat every headline number as a snapshot. METR's 2025 trial measured a 19% slowdown, and METR's own 2026 update says that result no longer reflects current tools while its new data is too compromised to replace it.[5][6]
- Build ROI from reclaimed capacity, incident cost avoided and earlier delivery, then subtract licenses, the learning dip and hidden drag such as CI compute and senior review time. Set thresholds from your own baseline, not from borrowed benchmarks.
How this article was put together. It's a synthesis of published research, vendor telemetry reports and framework documentation, all listed in the References. It contains no original benchmarks. Figures are quoted as the sources report them, and vendor-produced data is flagged as such. "DORA" here means DevOps Research and Assessment, not the EU Digital Operational Resilience Act.
Why activity metrics don't show delivery
By late 2025 AI assistance had become part of the normal toolkit. DORA's 2025 survey of nearly 5,000 technology professionals put adoption at about 90%.[1][2] The numbers that usually get reported for it are suggestions accepted, lines generated and active seats. All three describe activity.
Delivery is the whole path from an idea to working software in production, and writing code is one step on that path. Requirements, design, review, integration, testing and operating the result all sit around it. Make authoring faster while those other steps keep their old capacity and the queue doesn't disappear. It shows up somewhere else.
The telemetry points the same way. In Faros AI's 2025 analysis, developers on high-adoption teams completed 21% more tasks and merged 98% more pull requests, but PR review time rose 91% and organization-level delivery metrics didn't improve.[8][9] DORA's 2025 report frames the pattern as AI acting as an amplifier: it magnifies the strengths of high-performing organizations and the dysfunction of struggling ones.[1]
Local gains, in other words, don't convert to system gains on their own.
What the research says
No single study settles this, and the findings look contradictory until you compare what each one actually measured.
| Source | What it reported | Caveat |
|---|---|---|
| Peng et al. (2023), GitHub Copilot experiment[7] | Developers with the assistant finished the task 55.8% faster. | One well-defined task (an HTTP server in JavaScript), not a mature production codebase. Some authors are GitHub employees. |
| METR randomized trial (2025)[5] | 16 experienced open source developers on their own repos took 19% longer with AI (confidence interval +2% to +39%). They had predicted a 24% speedup and still believed afterwards they'd been about 20% faster. | Small sample and early-2025 tools. METR says these historical results no longer reflect current tools. |
| METR update (February 2026)[6] | A second study with 57 developers and 800+ tasks estimated that tasks took 18% less time with AI for returning developers (interval from 38% less to 9% more) and 4% less for new ones (interval from 15% less to 9% more). | METR calls this an unreliable signal. Many developers refused to work without AI or held tasks back, which biases the estimate. The study is being redesigned. |
| DORA reports (2024, 2025)[1][2][3] | 2024 linked adoption to lower delivery throughput and stability. 2025 linked it to higher throughput but still higher instability, and described AI as an amplifier. | Survey-based and correlational. Respondents skew toward teams already familiar with DORA practices. |
| Faros AI telemetry (2025, 2026)[8][9] | 2025: more tasks and PRs, longer reviews, bigger PRs, flat organization-level delivery metrics. 2026 (22,000 developers, 4,000 teams): throughput gains alongside more bugs, more incidents and longer review cycles. | Vendor-produced, correlational data. Faros sells engineering analytics. |
| McKinsey analysis[13] | Gains vary with task complexity and developer experience. | Consultancy analysis with its own methodology. |
Read together, the sources disagree less than the headlines suggest. A greenfield task with a clear spec behaves differently from a large, mature codebase. Self-reported speedups and measured speedups can diverge, which is the main lesson of METR's perception gap. And team-level outcomes depend on what happens after the code is written.
The practical conclusion is to measure your own system with telemetry, and to stay skeptical of any single figure, including the ones in this article.
Can you say whether cycle time from first commit to production got shorter or longer since AI rollout, using Git and CI data rather than a survey? If the only evidence is logins and developer opinion, you're reporting activity and sentiment.
Three metrics to drop first
Before adding anything, remove what misleads.
| Metric | Why it misleads |
|---|---|
| Suggestion acceptance rate | It records that a suggestion was kept at that moment, nothing more. In METR's 2025 trial developers accepted fewer than 44% of AI generations, and most reported making major changes to clean up what they kept.[5] |
| Lines of code added | Volume isn't value, and deletion is often the better change. Counting lines rewards duplication, and GitClear's analysis links AI-assistant adoption to rising churn and more copied code.[11] |
| Time spent in the tool | It can't tell productive use from going in circles. The SPACE framework makes the broader point that productivity can't be captured by activity counts or any single dimension.[12] |
If a developer reverted an AI-generated change after the tests failed, would any number on your dashboard move? If not, it can't tell a good suggestion from a bad one.
Stage 1: Development
The goal here is less mechanical work without making the codebase worse. Compare every metric below against the same teams' pre-rollout baseline, and against similar kinds of work.
Time to PR ready
Measure from first commit on a branch to the PR being marked ready. If assistants help with boilerplate, test scaffolding and routine CRUD, this should drop. If it falls while review time rises, the work moved rather than shrank.
PR size
Track the median and the 90th percentile of lines changed per merged PR. Faros reported PR size up 154% on high-adoption teams,[9] and DORA's AI capabilities model lists working in small batches among its seven practices.[1] Pick a soft ceiling from your own pre-rollout distribution instead of borrowing a number.
30-day churn
What share of merged code is edited, rewritten or deleted within a month? GitClear's analysis (a vendor dataset with its own method) reports rising churn and more duplicated code since assistants spread.[11] On AI-heavy repos, a rising line here is a prompt to look at edge-case handling and review quality.
Review wait and reviewer load
Measure time from PR ready to first review and to approval, plus the hours senior engineers spend reviewing. Faros found review time up 91%,[8] and a DX summary of the 2025 DORA findings describes code review becoming the dominant bottleneck when volume rises without better review systems.[14]
If junior output goes up and senior review hours go up with it, throughput may be flat even though the dashboard looks great.
Do you know how many hours per week your most senior engineers spend reviewing PRs, and whether that changed after rollout? If not, the cost of faster authoring may be landing where nobody is looking.
Stage 2: Deployment
Code waiting in a pipeline earns nothing. This stage shows whether AI-written code survives tests, scans and staging.
Lead time for changes
DORA defines it as the time from a code commit to running in production.[1] If you measure from merge instead, say so, because the number isn't comparable otherwise. A faster IDE with a flat lead time means the constraint is elsewhere: manual QA, approvals, flaky tests.
Mutation score
Line coverage shows which lines ran, not whether the tests would catch a bug, and generated tests can raise coverage cheaply. Mutation testing injects small deliberate faults and checks whether the suite fails. High coverage with a low mutation score means the tests aren't testing behavior.
CI first-pass failure rate
A sustained rise after rollout deserves investigation. Likely causes to check are broken interface contracts, mocks of things that don't exist, and type errors.
Security findings per PR
Pearce et al. tested GitHub Copilot on security-relevant scenarios and found about 40% of generated programs were vulnerable.[10] That study tested a 2021 version, and tools have changed a lot since, so use it as a reason to measure rather than as a current rate. Track static analysis findings per PR before and after rollout and keep them flat or falling.
Delete half the assertions in a well-covered AI-assisted module. Does the mutation score fall? If you only track line coverage and it stays green, the tests may be decoration.
Stage 3: Day-2 maintenance
The real test comes months after rollout. Saving authoring time means little if the code causes outages or nobody can maintain it.
Change failure rate and rework rate
The percentage of deployments that need a rollback, hotfix or incident, plus the share of deployments that were unplanned fixes. DORA's 2025 model groups these as instability metrics, and its 2025 findings still associate AI adoption with higher instability.[1][3][14] If either creeps up, creation is outrunning verification.
Failed deployment recovery time
This is worth testing in both directions. Code that was merged without much human understanding may take longer to diagnose under pressure. Assistants may also shorten recovery through log analysis, runbook lookup and incident summaries. Neither effect is guaranteed, so compare your incidents before and after.
Migration and refactoring speed
Framework upgrades, version bumps and security patches across many repos are a natural fit for agentic tools. Reported time savings vary widely by codebase, so measure it on your own repos with a before-and-after on comparable migrations.
Take your last five incidents. Was AI-assisted code involved, and could the on-call engineer explain the failing code without re-reading it from scratch? A pattern of "no" answers is worth acting on.
A simple scorecard
Define healthy and warning against your own baseline. The directions below are what to look for.
| Stage | Healthy | Warning |
|---|---|---|
| Development | Time to PR ready down, PR size steady or smaller, churn flat or falling, review load stable. | PR size growing, senior review hours climbing, churn rising on AI-heavy files. |
| Deployment | Lead time down, mutation score steady or up, security findings flat. | High coverage with a low mutation score, repeated CI breaks from type or mock errors. |
| Maintenance | Change failure and rework rates stable, upgrades getting faster. | More hotfixes, on-call engineers struggling with code they didn't write. |
Building an ROI model finance can check
The formula is simple: net ROI % = (value generated minus total cost) divided by total cost, times 100. The hard part is filling both sides honestly, and DORA's ROI guidance is built around that gap between engineering speed and financial impact.[4]
Total cost
Licenses are the visible part, so use current vendor pricing for your tool mix. Add usage-based API charges, indexing and security tooling. Then add the learning dip: DORA's guidance describes a J-curve, an early productivity dip while teams adapt, and recommends budgeting for it explicitly as a tuition cost.[4]
Value
Value comes from three places.
- Reclaimed capacity. Hours saved per week, times fully loaded hourly rate, times a reinvestment factor. No factor is universal, so set it conservatively, justify it, and show a low, mid and high case. Meetings, design and planning absorb much of what's saved. DORA's guidance stresses reinvesting reclaimed capacity, mainly by cutting rework, rather than assuming headcount savings.[4]
- Incident cost avoided. The drop in change failure rate times average cost per incident, plus recovery hours saved times outage cost per hour. This can be negative if instability rises.
- Earlier delivery. Revenue or business value gained because a milestone landed sooner. Hard to prove, and often the largest line.
Hidden drag
Subtract extra CI compute from higher commit volume and the senior time moved from design work to reading long PRs. Don't count the same saved hour in both the capacity line and the cycle-time line.
Could you defend your reinvestment factor to a skeptical finance partner in under a minute? "We assumed everything saved became output" won't survive the first question.
A 90-day measurement plan
- 1
Days 1 to 30: baseline
Pull Git and CI data for PR cycle time, review wait, churn, lead time and change failure rate before you change anything.
- 2
Days 31 to 60: run a proper comparison
Roll out in phases to comparable teams and compare delivery outcomes against the teams still waiting. Use telemetry, not logins or surveys, and watch for selection effects like the ones METR hit.
- 3
Days 61 to 90: add quality brakes
Set soft limits on PR size, put mutation testing and security scans in CI, and reward small modular changes over output volume.
The compliance angle
For companies selling into regulated markets, this measurement work doubles as evidence. SOC 2 audits examine change management, and the EU Cyber Resilience Act (Regulation (EU) 2024/2847) requires manufacturers to design and develop products with digital elements against essential cybersecurity requirements. Per-PR security findings, review records and change failure rate are the kind of artifacts those conversations ask for.
The legal text is on EUR-Lex. For how we approach it in practice, see our Cyber Resilience Act guidance.
Key concepts covered
- AI coding tools
- Engineering metrics
- DORA
- Lead time for changes
- Change failure rate
- Rework rate
- Mutation testing
- PR size
- Code churn
- Review load
- ROI model
- J-curve
Frequently asked questions
What is the best metric for measuring AI coding tool impact?
There isn't a single one. The SPACE framework argues that productivity can't be reduced to one dimension, so pair a speed metric such as lead time for changes with a stability metric such as change failure rate and read them together. If speed improves while stability slips, the tools are moving work around instead of removing it.
Why is suggestion acceptance rate a poor metric?
It only records that a suggestion was kept at that moment. In METR's 2025 trial, developers accepted fewer than 44% of AI generations and most reported making major changes to clean up what they kept, so the rate says little about time saved or whether the result was right.
How long before AI coding tools show real results?
DORA's ROI guidance describes a J-curve: an early productivity dip while teams learn, then gains, and it recommends budgeting for that learning cost. Delivery metrics move more slowly than authoring metrics, and maintenance metrics such as change failure rate move slowest, so plan in months and compare against your own baseline.
How do you run a fair comparison between teams with and without AI tools?
Roll out in phases to comparable teams and compare delivery outcomes such as cycle time, review wait, change failure rate and rework against the teams still waiting. METR's 2026 update found that some developers refuse to work without AI, which biases strict no-AI control groups, so a staged rollout is usually more realistic than a permanent holdout.
Should we cap pull request size when using AI assistants?
A soft limit is reasonable. Faros reported PR size growing sharply on high-adoption teams, and DORA's AI capabilities model includes working in small batches. Set the limit from your own pre-rollout distribution, make it a CI warning first, and tighten it only if review times stay high.
Do AI coding tools make code less secure?
They can introduce vulnerabilities. A 2022 study of Copilot by Pearce et al. found about 40% of generated programs in security-relevant scenarios were vulnerable, though it tested a 2021 version and tools have changed since. Track static analysis findings per PR before and after rollout instead of relying on an old figure.
Not sure what your AI tools are really doing to delivery?
We help engineering leaders set up the baseline, the CI guardrails and the measurement model, so the next budget conversation is about shipped outcomes. If you'd like a second pair of eyes on your numbers, talk to us.
Talk to Stonetusker about your engineering metricsReferences
- DORA, Google Cloud. 2025 State of AI-assisted Software Development report. https://dora.dev/dora-report-2025/
- Google. How are developers using AI? Inside our 2025 DORA report (September 2025). https://blog.google/innovation-and-ai/technology/developers-tools/dora-report-2025/
- InfoQ. DORA Report Finds AI Is an Amplifier in Software Development, But Trust Remains Low. https://www.infoq.com/news/2025/09/dora-state-of-ai-in-dev-2025
- DORA, Google Cloud. ROI of AI-assisted Software Development. https://cloud.google.com/resources/content/dora-roi-of-ai-assisted-software-development
- METR. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (July 2025). https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
- METR. We are Changing our Developer Productivity Experiment Design (February 2026). https://metr.org/blog/2026-02-24-uplift-update/
- Peng, Kalliamvakou, Cihon, Demirer. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. https://arxiv.org/abs/2302.06590
- Faros AI. The AI Productivity Paradox Report 2025. https://www.faros.ai/blog/ai-software-engineering
- Faros AI. Research hub, including the AI Engineering Report 2026. https://faros.ai/research
- Pearce et al. Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions. IEEE S&P 2022. https://www.computer.org/csdl/proceedings-article/sp/2022/131600a980/1FlQxERjKCs
- GitClear. Coding on Copilot: data shows AI's downward pressure on code quality. https://www.gitclear.com/coding_on_copilot_data_shows_ais_downward_pressure_on_code_quality
- Forsgren et al. The SPACE of Developer Productivity. ACM Queue. https://queue.acm.org/doi/10.1145/3454122.3454124
- McKinsey. Unleashing developer productivity with generative AI. https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/unleashing-developer-productivity-with-generative-ai
- Engineering Enablement (DX). DORA's 2025 research on the impact of AI. https://newsletter.getdx.com/p/doras-2025-research-on-the-impact
- European Parliament and Council. Regulation (EU) 2024/2847 (Cyber Resilience Act), EUR-Lex. https://eur-lex.europa.eu/eli/reg/2024/2847/oj
