How to detect quality regressions in AI agents in production
Image generated by the Inferama Editorial Team
01

Availability does not mean quality

An agent can respond to every request, maintain acceptable latency, and log no infrastructure errors while still doing the task badly. It might omit a condition, consult an outdated source, invoke an unsuitable tool, or present an uncertain result as reliable. A dashboard that only measures availability, errors, response time, and resource use will not necessarily reveal these problems.

Separate three questions. Is the service still running? What steps did the agent take? Did the result meet the process’s purpose and rules? Technical metrics help answer the first; traces help reconstruct the second; task evaluations examine the third. None of these layers replaces the others: a correct answer does not prove that the route was authorized, and a complete trace does not prove that the result was useful.

Quality evaluation should start with the specific use case and the consequences of getting it wrong. For an information-retrieval task, the answer may need to be supported by a current record. For a transactional action, parameter validation and authorization may be mandatory. An aggregate metric can conceal important failures in a task category, language, user group, or stage of a workflow.

Continuous evaluation can help identify changes in an agent’s behavior after deployment. It does not guarantee that every failure will be detected, nor does every difference in a score automatically constitute a real regression. Interpretation depends on the test set, the evaluator, the configuration being compared, and measurement uncertainty.

Three layers of control

LayerQuestion it answersExample signals
Technical healthIs the service operational?Availability, errors, latency, retries, and resource use.
BehaviorWhat did the agent do to complete the task?Retrieved sources, invoked tools, parameters, and escalations.
Functional qualityWas the result correct and acceptable for the process?Task completion, rule compliance, relevant sources, and authorized actions.
02

Define what counts as a well-completed task

A quality metric is useful only if it corresponds to a condition that matters for the intended use. “Good answer” is too vague to guide an alert. Specify what the agent must produce, which actions it must not take, when it should abstain, and what evidence can verify the result. Define these criteria before looking at the results to reduce the risk of tailoring the rule to what the system has already done.

Combine deterministic checks with interpretive evaluation where appropriate. A conventional validator can check whether a required field exists, an identifier belongs to an authorized set, or an output matches a schema. It cannot, on its own, decide whether an explanation faithfully reflects an ambiguous document. Human rubrics or model-based evaluators can help assess meaning, but their judgments also need to be checked against human references.

Overall task success is not always enough. Track critical failures, incomplete cases, appropriate abstentions, human interventions, and actions that exceed the agent’s granted permissions separately. An average success rate can improve while a control that should not be offset by easy wins gets worse.

The examples below show how to translate goals for different types of agents into verifiable criteria. They are not universal thresholds: each team should adapt them to its data, rules, tools, and level of risk.

Examples of task-specific criteria

Agent typeCriteria that can be checkedWhat needs interpretation or additional review
Coding assistantDo the defined tests pass? Does the agent respect file or dependency constraints? Does the output pass the compiler or analyzer?Whether the solution is well designed, maintainable, and secure; passing a test does not rule out every defect.
Document researchDo claims cite retrieved documents? Are the version and date relevant? Does the agent abstain when support is missing?Whether the citation actually supports the claim or the synthesis omits an important qualification.
Customer supportDoes the agent identify the issue and apply the current policy? Does it avoid disclosing data beyond its permissions? Does it escalate cases that need a person?Whether the response meets the customer’s need in an appropriate tone or correctly interprets an exception.
Agent with toolsDoes it select an authorized tool? Are its arguments valid? Does it require approval before a sensitive action?Whether the trajectory was proportionate and whether the action was justified by the full context.
03

Evaluate interactions without interrupting the service

A shadow-evaluation architecture analyzes a production interaction without changing the response the user receives or executing a second action in the business system. For example, it can evaluate a recorded trace or a sanitized copy of a conversation after the task ends. Evaluation should not re-run external actions: a second call to a write-enabled tool could duplicate an operation or cause unwanted effects.

Sampling can reduce the volume reviewed, but there is no universally appropriate sampling rate. It depends on how often cases occur, how much results vary, how sensitive the process is, and what change the team wants to detect. A random sample helps represent the traffic being studied; a sample targeted at incidents can expose rare failures but cannot, on its own, estimate how often they occur across the entire population. Keeping these purposes separate prevents a search for difficult cases from being mistaken for a representative measurement.

Before scoring real interactions, limit capture to what is necessary. Traces may contain messages, documents, tool arguments, and personal data. Apply data minimization, filtering, access controls, and suitable retention from the outset. If an evaluation sends content to another model for judging, also review what data leaves the application, where it is processed, and which terms apply to the configured service.

Microsoft Foundry documents evaluation runs with test sets and describes continuous monitoring in production. That documentation explains capabilities of the platform, but does not establish a universal sample size, or prove that every shadow-evaluation design is random, harmless to privacy, or sufficient to detect a particular regression.

Shadow-evaluation workflow

  1. 01Define which kinds of interactions you want to observe and why: estimate typical performance, investigate a failure, or monitor a critical control.
  2. 02Select a sample using a documented method; keep representative samples separate from samples targeting rare or high-risk cases.
  3. 03Remove or transform unnecessary fields, and restrict who can access the original data and results.
  4. 04Evaluate the outcome and, where relevant, the recorded trajectory. Do not repeat the agent’s external actions in production.
  5. 05Record the agent version, instructions, model, tools, evaluator, criteria, and date so later comparisons can be interpreted.
04

Validate evaluators before relying on their scores

A model-based evaluator can classify responses using a rubric and process more cases than an exhaustive manual review. However, it can also miss an exception, favor longer answers, or score differently depending on language and phrasing. Its score is a signal that needs calibration, not ground truth.

Prepare examples whose ratings have been reviewed by people familiar with the process. Include correct, incomplete, incorrect, and borderline cases; compare the automated decision with the reference and examine false passes and false failures separately. If evaluator errors affect an important control, revise the rubric, change the method, or require human review before using the score to make an operational decision.

Deterministic rules are preferable when a property can be checked exactly: a required field, a structured format, an existing identifier, a numerical limit, or an invocation of a prohibited tool. But syntactically valid output can still contain false information. A high score from a judge also does not prove that the agent used authorized sources or that its result is correct in every case.

Version the criteria and the evaluator. A change to the rubric, judge model, or configuration can alter scores even if the agent has not changed. For a meaningful comparison, record the full configuration and, where possible, evaluate the versions being compared using the same test set and conditions.

Choosing an evaluation method

Property to verifySuitable initial methodLimit to keep in mind
Format, fields, and rangesDeterministic validatorChecks structure or limits, not necessarily meaning.
Permissions and tool callsSystem rules, logs, and controlled testsLogs help reconstruct calls; authorization must be enforced by the relevant system.
Relevance of a synthesis or explanationHuman rubric or calibrated model-based evaluatorThe result depends on the quality of the rubric and evaluator.
Correctness in a high-impact caseExpert reference and human reviewAutomated evaluation can assist but does not replace approval responsibility.
05

Set alerts and respond proportionately

A useful alert starts with a task-appropriate baseline and an action criterion defined before an incident. Compare periods and versions only when conditions are sufficiently comparable; a change in traffic, data, tools, or configuration may explain a difference. If the volume is small or normal variation is high, an observed deviation may not be enough to conclude that a regression occurred.

Do not use a single aggregate score as an approval gate. Separate non-negotiable controls—such as unauthorized access, an unauthorized write, or the loss of a required validation—from adjustable goals such as latency or stylistic quality. Improvement in one area should not conceal deterioration in a critical control.

Every signal needs a possible response and an owner. Depending on the risk, the team can investigate a sample, increase human review, restrict a tool, limit automation, or revert to an earlier version. A pause or rollback must be implemented through verifiable operational controls; an alert alone does not stop the agent or prevent further actions.

Set thresholds with the process owner and consider the cost of a false alarm against the cost of missing a failure. Statistical significance depends on the metric, baseline rate, sampling design, and assumptions used. Without those elements, it is not valid to claim that a fixed sample size will detect a specified change with universal confidence.

Responding to a regression signal

  1. 01Confirm that the alert concerns a relevant change and that the sample and conditions support comparison with the baseline.
  2. 02Review specific cases, traces, and configuration; distinguish an agent regression from a change in data, external systems, or evaluation criteria.
  3. 03Request human review for ambiguous or high-impact cases and classify severity by consequence and reversibility.
  4. 04Apply a proportionate measure: continue monitoring, limit autonomy, disable a tool, or pause the affected workflow.
  5. 05Verify the correction with regression tests before restoring or expanding use, and document the operational outcome.

Signal and possible initial response

SignalPossible initial responseIt does not establish by itself
More incomplete responsesReview cases and compare by task type.That the whole system has worsened equally.
Prohibited tool or out-of-range parameterContain the affected capability and review authorization controls.That changing only the prompt will resolve the issue.
Drop in an automated scoreCheck the evaluator, sample, versions, and human references.That the agent is less correct without checking examples.
Insufficient tracesLimit hard-to-audit operations and fix instrumentation.That a complete trace guarantees a correct decision.
06

Turn failures into regression tests

An interaction that reveals a problem can help check that the same failure does not recur. Before adding it to a test set, reduce the example to what is necessary, remove identifying information where appropriate, and retain the functional context that explains why the result was wrong. Not every production record should automatically become test data.

Each case should include a defined task, the expected result or evaluation criteria, relevant conditions, and the version it applies to. If the case involves a tool, also record which call was valid and which should have been avoided. The expected behavior can describe a verifiable property without requiring one exact text response when several responses are acceptable.

Keep development cases separate from cases reserved for checking changes. If a team repeatedly tunes the agent on the same examples, it may improve on that set without solving the broader problem. Add new representative cases, review older data, and document why each test is retained.

Changes to the model, instructions, tools, permissions, retrieved sources, and business rules can alter behavior. Running a regression set after a change helps compare results, but does not prove that the system is stable under every production condition. Evaluation must remain tied to risk and real use.

07

Practical criteria for continuous evaluation

Start with a bounded task and a failure whose consequences are understandable. Define expected quality, controls that cannot be offset by other results, and the review procedure before connecting evaluation to real traffic. A more limited initial setup may be preferable to logging everything without a clear purpose.

Combine infrastructure metrics, traces, and functional checks. Use deterministic rules for exact properties, human review for difficult judgments, and model-based evaluators only after checking them against suitable references. Keep the signals distinct: an automated score is a measurement; a confirmed incident requires interpreting the case and its context.

Treat privacy and security as part of the design. Minimize captured data, limit retention and access, and determine whether external evaluators receive interaction content. Do not retain complete conversations simply because they might be useful in the future; the need should serve a specific purpose and applicable obligations.

Continuous evaluation adds value when it helps explain what changed, which condition failed, and what operational decision follows. It does not replace deterministic controls, pre-deployment tests, human oversight for sensitive actions, or review of system changes. The goal is to detect useful signals with explicit limits and respond before unacceptable behavior has a broader impact.

Operational checklist

QuestionPractical criterion
Is the task defined?There is a verifiable outcome, edge cases, and conditions for abstaining or escalating.
Is the process evaluated as well as the text?The trace allows review of relevant sources, tools, parameters, and approvals.
Is telemetry protected?Data is minimized, and access, uses, and retention periods are defined.
Has the evaluator been checked?Its errors are compared with human decisions or verified references.
Does an alert lead to action?There is an owner, a review path, and an option to limit or stop the workflow.
Are changes checked?Regression tests run with recorded versions and criteria.
08

Keep exploring

Sources consulted

03

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.

Submit a correction