AI coding assistants: what the controlled studies and large surveys actually measured

Four studies now shape the debate about AI coding assistants, and they appear to disagree. One found experienced developers slower, one found engineers faster, and two large surveys report that most developers feel more productive. They are not contradicting each other so much as measuring different things. Reading them side by side shows what each can and cannot tell a buyer.

The randomized trials

METR (July 2025). METR ran a randomized trial with 16 experienced open-source developers working on 246 real issues in large repositories, averaging "22k+ stars and 1M+ lines of code," that they had contributed to for years ^[1]. When AI was allowed, developers chose their own tools, mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet ^[1]. Result: they took 19% longer to finish issues with AI than without ^[1].

The perception gap is the most striking part. Before the work, developers expected AI to speed them up by 24%. Afterward, despite the measured slowdown, they still believed it had sped them up by 20% ^[1].

METR is careful about scope. It does not claim its developers or repositories represent most software work, it cannot rule out learning effects beyond about 50 hours of Cursor use, and it says the result does not show AI is unhelpful for less experienced developers or unfamiliar codebases ^[1].

Google (arXiv, October 2024). Google-affiliated authors ran a randomized controlled trial with 96 full-time Google software engineers on a complex, enterprise-grade task ^[2]. Their best estimate was that AI features sped the task up by about 21%, though they describe the confidence interval as large ^[2].

The large surveys

DORA 2025 (September 2025). Google Cloud's DORA programme surveyed nearly 5,000 technology professionals. It reports 90% AI adoption, a 14% increase on the prior year, with a median of two hours a day spent working with AI ^[3]. More than 80% said AI had enhanced their productivity, and 59% reported a positive influence on code quality ^[3]. Trust is more guarded: 24% have a great deal or a lot of trust in AI output, while 30% have a little or none ^[3].

On delivery, the report links AI adoption to higher throughput, reversing last year's finding, but InfoQ's summary says software delivery instability continues to rise, and researchers found no sign that speed offsets it through fast failure and repair ^[4]. Google's own summary describes AI as a "mirror and a multiplier" of the organization around it ^[3].

Stack Overflow 2025. The survey drew over 49,000 responses from 177 countries. It reports that 84% of respondents use or plan to use AI tools, up from 76%, and 47.1% use them daily ^[5]. Favorable sentiment is 59.7%, down from more than 70% in 2023 and 2024 ^[5]. Only 3.1% highly trust the accuracy of AI output; 66% cite solutions that are "almost right, but not quite," and 45.2% say debugging AI-generated code takes more time ^[5].

Why the results differ

What was measured. METR and Google timed real tasks. DORA and Stack Overflow asked people how they feel. METR's own result shows why that matters: developers believed they were faster while the clock said otherwise ^[1]. Perception data is useful for adoption and morale, but it is not output data.

Task and codebase. METR's developers worked in mature repositories they knew deeply ^[1]. Google's engineers did one complex task in an internal environment ^[2]. Different work, different baseline knowledge, different results.

Tools and date. METR tested early-2025 tools ^[1]. Google's trial was submitted in October 2024, so it reflects older tooling still ^[2]. Neither speaks for tools shipping this autumn.

Sample. 16 developers and 96 engineers are small samples, and survey respondents are self-selected. Neither design settles the question for your teams.

Metric. DORA separates throughput from instability ^[4]. A faster pull request that raises rework is not a gain, and a single speed number hides that.

What this means for engineering leaders

Do not scale on any one of these studies, in either direction. Measure in your own environment first:

1. Pick the metric. Choose a delivery measure such as cycle time or lead time, paired with a quality measure such as change failure rate or rework. DORA's throughput-versus-instability split is a useful template ^[4].

2. Baseline it. Record the metric for several weeks before the tool arrives, so you are not comparing against memory.

3. Pilot with a comparison group. Roll out to a subset of teams and compare them with similar teams on comparable work, as METR and Google did with randomization ^[1][2]. Track self-reported productivity alongside, and expect the two to diverge.

4. Segment the results. Report by task type and developer experience, since the studies suggest both matter.