Scope the use case and potential harm
Before deciding how an automation should respond to an error, define the task’s boundaries and the potential impact of failure. Not all errors are equally serious: malformed text output is a validation problem, while carrying out an action with external effects—such as sending an email or changing a database record—poses a greater operational risk.
The key distinction is between generation failures, which affect the quality of a model’s output, and execution failures, which affect the system’s ability to complete the task. Separating them helps you apply proportionate controls: strict validation for generated output, and state checks that prevent repeated effects when carrying out an action. This guide focuses on what the workflow should do when it cannot complete a task, returns a questionable result, or loses a dependency—not on choosing a pilot task or evaluating a model’s general quality.
Classify failures by impact
Use this table to distinguish generation, dependency, and execution failures and identify a suitable primary control for each.
| Failure type | Example | Potential impact | Primary control |
|---|---|---|---|
| Generation | Text in an incorrect format | Low | Schema validation |
| Dependency | An external API does not respond | Medium | Bounded retry with backoff |
| Execution | An action is performed twice | High | Check the prior state |
| Uncertain outcome | The action may have completed, but confirmation is missing | Potentially high | Abstain and verify before acting again |
Build a catalog of observable states
An effective recovery policy requires each failure condition to be detectable through an objective, reproducible signal. Avoid relying on subjective impressions such as “the response seems doubtful.” Define conditions that can be checked against a validation result, a status, a timeout, or another observable signal.
Include the states that matter for this workflow: invalid or incomplete output, a request that times out, a dependency that is unavailable, an ambiguous result, and an action whose completion status is unknown. Map every state to a defined response. If a condition cannot be detected reliably, do not pretend the workflow can handle it automatically; specify how it reaches a person instead.
Choose a response for each state
For every observable failure state, define a default response: retry within a limit, abstain and leave the task unresolved, send it for human review, or return it to the manual procedure. The choice depends on whether the action is reversible, the consequences of an error, and what is known about the action’s current state.
Retries can be appropriate for transient dependency failures, provided their number and duration are capped. Abstention is safer when the action’s status is uncertain: do not perform the final action again until its status has been checked. Human review is appropriate when the consequences are significant, the outcome is ambiguous, or the action is difficult to reverse. For a manual fallback, name the person or role responsible and specify what information they need to take over.
Recovery decision matrix
A starting template for mapping observable states to responses and closure conditions. Set retry limits for your own system and verify them in testing.
| Failure state | Default response | Condition for closure |
|---|---|---|
| Request timed out; no action was initiated | Retry within a defined limit | Successful response or escalation |
| Output is invalid or incomplete | Abstain from the consequential action | Human review or corrected output |
| Dependency is unavailable | Retry within a defined limit, then fall back | Dependency restored or manual handoff |
| Action outcome is ambiguous | Check the destination state; do not repeat the action yet | Confirmed completion or safe, authorized retry |
| Action is high-impact or hard to reverse | Require human review or approval | Explicit decision and recorded outcome |
Prevent duplicate effects with idempotency and state checks
A major recovery risk is repeating an action that may already have taken effect. If an automation retries after an interruption, it could send the same email twice or create duplicate records. Distinguish repeating a read-only query from repeating an action that changes something outside the workflow; the latter requires additional safeguards.
Where possible, design consequential actions to be idempotent: repeated requests should not produce additional effects beyond the intended result. If that is not possible, check the destination system’s current state before retrying. Treat that check as a required transition in the recovery flow, not as an optional convenience. If the state cannot be confirmed, abstain and hand the task to an authorized person rather than guessing.
Check the prior state before retrying an action
Use this sequence when an action may have been interrupted or its result is uncertain.
- 01Identify the action that failed or was interrupted, and determine whether it can have external effects.
- 02Query the destination system or another reliable record to find out whether the action completed.
- 03If completion is confirmed, mark the task resolved without repeating the action.
- 04If non-completion is confirmed, retry only if the action is authorized and the retry is safe; use a unique request or operation identifier where supported.
- 05If the state remains uncertain, abstain and route the task to the designated person.
- 06Record the state check and the resulting decision in the workflow’s existing audit record.
Design the state machine and handoff
Represent the recovery policy as explicit states and transitions so it can be implemented and tested. A task might move from “received” to “processing,” then to “completed,” “retry eligible,” “awaiting review,” or “manual fallback.” For each transition, document its entry condition, the action permitted, the owner, and the condition for closing the task.
Pay special attention to recovery after interruption. A task should not disappear between automated and manual handling, and a person should not receive a case without enough context to act. Define what information travels with the handoff, who is responsible for the next step, and how the workflow records that responsibility. Keep the permission boundary explicit: recovery must not authorize an action that the original task did not permit.
State-transition worksheet
Adapt this worksheet to the task and its destination systems. A named owner can be a defined team or role, provided responsibility is unambiguous.
| Current state | Entry condition | Permitted response | Next state or owner | Closure condition |
|---|---|---|---|---|
| Processing | Task accepted and authorized | Validate output and dependencies | Completed, retry eligible, or awaiting review | The outcome is confirmed or handed off |
| Retry eligible | A transient failure is detected and no consequential action is known to have occurred | Retry within the documented limit | Processing or awaiting review | Success or retry limit reached |
| Awaiting review | Output is invalid, result is ambiguous, or a defined escalation condition is met | No consequential action without authorization | Named reviewer or operating role | Reviewer records a decision |
| Manual fallback | Automation cannot continue safely or a dependency remains unavailable | Follow the documented manual procedure | Assigned person or team | Manual outcome is recorded |
Test with induced failures and acceptance criteria
A recovery policy is not ready simply because its normal path works. Test it with controlled failures: an unavailable dependency, malformed output, a delayed response, an ambiguous action result, and an interruption during recovery. Confirm both that the intended recovery works and that the workflow abstains or escalates when it cannot safely determine what happened.
Make acceptance criteria observable and verifiable. Check whether the failure was detected, whether the configured retry limit was respected, whether the correct response ran, whether any consequential action was duplicated, and whether a named owner received an actionable handoff. Do not treat these checks as a guarantee of safety or legal compliance; they are operational tests of the specified workflow.
Publish a reusable matrix and readiness checklist
Keep the failure matrix editable and specific to the workflow. It should name each detectable state, the signal that identifies it, the allowed response, the retry limit if applicable, the handoff owner, and the condition for closure. Include a separate check for cases where the outcome of an external action is unknown; these are not ordinary transient errors.
Before moving from testing to controlled use, confirm that every in-scope failure state has a reproducible detection method, each retry is bounded, and a retry cannot repeat an action without the required state check. Verify that abstention and human review have named owners, the manual fallback is documented, and induced-failure tests cover both recovery and safe non-action. These criteria support a deployment decision; they do not establish a universal guarantee or replace independent review where it is needed.
Acceptance checklist
Use these checks to decide whether the specified recovery behavior is ready for controlled use.
| Acceptance check | Pass condition |
|---|---|
| Observable failure states | Every in-scope state has a reproducible detection signal |
| Bounded retries | The maximum attempts and stopping conditions are documented and tested |
| No duplicate effects | Action retries require idempotency or a verified prior-state check |
| Safe handling of uncertainty | Unknown action status leads to abstention and verification, not a blind retry |
| Accountable handoff | Every escalation has an identifiable owner and sufficient context |
| Manual fallback | The return to the manual procedure is documented and tested |
| Induced-failure coverage | Tests cover dependency loss, invalid or delayed output, ambiguity, and interruption |
| Permission boundary | Recovery does not execute an action that was not authorized |
Open questions
- The primary evidence supplied is Azure Well-Architected documentation. It offers relevant architectural guidance on reliable automation, validation, approvals, and recovery, but its scope does not establish a universal policy for AI agents or all automated workflows.
- The secondary source provides general context about small-business automation; it does not substantiate technical implementation details for idempotency or AI recovery.
- Retry limits, backoff timings, escalation thresholds, and acceptance criteria must be set for the particular workflow and verified in its environment; the examples here are not universal safety guarantees.
- This guide describes operational design recommendations, not legal obligations or a guarantee that an automation is safe. High-impact uses may require independent technical, security, or legal review.
Keep exploring
Sources consulted
Corrections and transparency
If you spot incorrect or outdated information, send us a correction with the page and source we should review.
Submit a correction