Some defects can disappear simply because we change what we measure. Not because we have fixed them, prevented them or detected them earlier. They simply stop appearing on the dashboard because the organisation has started tracking something else: test cases generated, automation rates and other activity metrics.
AI can amplify this effect because it allows organisations to improve activity metrics rapidly. But software quality needs to answer a different question: what is actually reaching production?
The paradox is that, even as AI adoption in testing increases, defect rates do not necessarily fall. In some cases, they even rise.
This is reflected in the State of Digital Quality in Functional Testing 2026, which reports that more than 92% of respondents use AI in their testing processes, compared with 62% the previous year. Yet 29% report an increase in either the number or severity of functional defects, while 15% say both have increased.
We are generating more tests, faster, yet the findings also point to a deterioration in quality. Of course, correlation does not imply causation, and the study does not establish that AI is responsible for this deterioration. It does, however, challenge one assumption: increasing testing capacity does not necessarily reduce defects.
We can improve every testing activity metric without improving the outcome that really matters: the quality of the software we deliver.
More tests do not necessarily mean better risk coverage
How can organisations generate more tests while also encountering more defects, or more severe ones? To understand this, we first need to distinguish between increasing testing capacity and improving testing effectiveness.
According to the report, the most common uses of AI in testing are generating test cases (65%), writing automation scripts (62%) and identifying coverage gaps (48%). These are useful capabilities, but their results are typically measured through activity metrics: test cases generated, automation coverage, completed test executions or reduced regression testing time.
As we have seen, however, activity and effectiveness are not the same thing. A test suite can grow by adding variations of scenarios that are already well understood while continuing to overlook important risks. If a risk is missing from the requirements, has never been explicitly identified or depends on real user behaviour, generating more tests does not guarantee that it will be covered.
We must also remember that AI is accelerating software development itself, increasing the volume of changes reaching QA. The report notes that 79% of respondents perceive a moderate or significant impact from AI in development, compared with 72% who report the same impact in testing.
The question, then, is not simply whether testing can execute faster. It is whether its ability to understand and cover risk is keeping pace with the rate of change.
If it is not, an organisation may increase automation and shorten regression testing cycles while allowing more defects to pass through its quality controls.
Five metrics that reveal whether quality is really improving
If we want to know whether we are genuinely protecting the product, activity metrics are not enough. We need to measure which defects escape our controls, the impact they have and the effort required to resolve them.
That means looking beyond test execution. Rather than simply adding more metrics to the dashboard, we need to answer five questions.
1. How many defects escape?
The defect escape rate measures the proportion of defects detected after the stage in which they should have been caught, particularly those that reach production.
2. How severe are they?
Defect counts need to be weighted by impact. A minor defect is not equivalent to a critical service outage, financial loss or exposure of sensitive information.
3. How much rework do they create?
Rework includes defect correction, investigation, retesting, coordination and redeployment. If the time or cost saved through AI reappears here, the improvement is only partial.
4. What costs do they generate?
The cost of poor quality captures the technical, operational and financial consequences of defects, from support activities to service unavailability and opportunity costs.
5. How do they relate to the software delivered?
Measuring quality per functional unit helps us determine whether we are finding more defects because we are delivering more software or because the quality of what we deliver is declining.
By answering these five questions and examining how the metrics relate to one another, we can establish whether testing is reducing risk or merely increasing activity.
From measuring testing activity to governing quality
However much AI expands our capacity to generate, execute and analyse tests, deciding what needs to be tested, what constitutes sufficient evidence and which risks are acceptable still requires human judgment.
The report reinforces this point: 86.1% of respondents consider human involvement in functional testing extremely important, particularly for providing qualitative feedback, conducting peer reviews and designing tests around real user behaviour.
Automation will continue to advance, but our ability to control its outcomes must advance with it.
To establish whether quality is genuinely improving, we need to connect the software delivered with the defects that reach production, their severity, the rework they generate and the cost of poor quality. Otherwise, we may simply be shifting effort into incident management and subsequent corrections.
At LedaMC, we help organisations establish comparable benchmarks to assess software quality, identify where quality is being lost and determine where to intervene, from requirements and testing strategy through to vendor management.
That is the difference between measuring for reporting and measuring for governance. More data does not necessarily mean more value. What we need are the metrics that show us where to act and whether our decisions are delivering results.
