Agent Failure, Retry & Reconciliation

An agent timeout is not a receipt saying “nothing happened.” A model call can time out after a tool request was submitted; a tool can time out after the remote system committed a side effect; a run can exceed its turn limit before any tool is called. Those incidents have different recovery implications.

The artifact for this lesson is Agent Failure, Retry & Reconciliation Ledger v1. Its job is to make failure phase, side-effect certainty, retry eligibility, bounded attempts, reconciliation and escalation reproducible from visible evidence.

First rule: an error is evidence of failure, not proof of zero side effects.

Route orientation and owned boundary

AGN-006 owns failure classification, retry control and reconciliation. AGN-005 owns memory boundaries; AGN-007 owns cost metering. This lesson may record retry attempts, but it does not decide memory retention or monetary allocation.

Foundation learners use the field map and worked scenario. Experienced learners may skip definitions, but they must pass the same side-effect, idempotency, attempt-budget, dependency-health and reconciliation gates.

Mental model: failure, uncertainty, recovery

Keep these layers separate:

1. Failure evidence — what failed, where, and with what observable error.
2. Side-effect certainty — not started, confirmed absent, confirmed applied, or unknown after ambiguity.
3. Retry control — eligibility, reason, attempt budget, attempt number, backoff and dependency health.
4. Safety control — idempotency key or a bounded compensating/reconciliation path for side-effecting work.
5. Reconciliation — compare observed state with an authoritative state before declaring success after ambiguity.
6. Escalation/final disposition — success, safe retry, hold, compensate, escalate, or bounded stop.

A later successful attempt does not rewrite the evidence from an earlier ambiguous attempt.

Foundations: current FACT versus local DESIGN

Current OpenAI Agents SDK documentation distinguishes failures such as maximum-turn exhaustion, model-call timeout, tool timeout and model-behavior errors. Tool timeout behavior can also be configured either as a model-visible error result or as a raised exception. These are runtime facts, not business retry policy.

AWS guidance on retries warns that retries can add load to a failing dependency, recommends bounded retries with backoff/jitter, and highlights idempotency for side-effecting operations.

MAKNOW's exact failure taxonomy, attempt budgets, reconciliation states and escalation triggers are local design. External documentation cannot declare a synthetic business operation safe to replay.

Deep explanation of the ledger

The critical contract has 18 dimensions: run_id, attempt_id, failure_class, failure_phase, error_evidence_ref, side_effect_state, idempotency_key, retry_eligible, attempt_budget, attempt_number, backoff_policy_ref, dependency_health, reconciliation_required, authoritative_state_ref, reconciliation_state, human_escalation_required, final_disposition, and audit_ref.

Important gates:

• A side-effect state of UNKNOWN_AFTER_TIMEOUT cannot be converted to “safe retry” merely because an exception was raised.
• Side-effecting retry needs an idempotency key or a bounded compensating/reconciliation control.
• attempt_number cannot exceed attempt_budget; exhausted attempts stop automated recovery.
• A degraded dependency makes immediate retry unsafe unless a governed backoff/retry decision is explicitly supported.
• If reconciliation is required, an authoritative state reference and a resolved reconciliation state are mandatory before success.
• final_disposition=SUCCESS is invalid while side effects, authoritative state or reconciliation remain unknown.

Worked examples

1. Timeout after a tool request

A synthetic order-adjustment request is sent, then the model call times out. Side effect is UNKNOWN_AFTER_TIMEOUT. The same idempotency key is available, but the ledger does not immediately retry. It queries a synthetic authoritative order record and finds the adjustment already applied. Reconciliation becomes MATCHED_APPLIED; the final disposition is RECONCILED_SUCCESS_NO_RETRY.

2. Invalid shortcut: retry because timeout happened

Keep side_effect_state=UNKNOWN_AFTER_TIMEOUT but set retry_eligible=true and remove the idempotency key. The case is blocked. Timeout did not prove that the prior side effect was absent.

3. Transfer: max-turn before tools + degraded dependency

A second run exceeds max turns before any tool invocation, so side effect is NOT_STARTED. That removes the original side-effect ambiguity, but the downstream dependency is degraded. Recompute retry eligibility and backoff from this new evidence. Do not copy the baseline “reconcile first” decision.

Failure modes

• Exception oracle: treating a timeout/error as proof that nothing happened.
• Blind retry: retrying an ambiguous side-effecting operation without idempotency or reconciliation.
• Budget overflow: continuing after the bounded attempt budget is exhausted.
• Hot-loop retry: retrying immediately while a dependency is degraded.
• Success overwrite: hiding an unresolved earlier attempt behind a later success.
• Authority-free reconciliation: declaring matched state without an authoritative state reference.
• Compensation fiction: naming a compensation action without evidence it is applicable or approved.
• Baseline oracle: reusing recovery logic after the failure phase or dependency health changes.

Alternatives and trade-offs

A failure policy can be fail-fast, retry, reconcile, compensate, or escalate. No option is universally best. Read-only operations often have lower side-effect risk; writes need stronger controls. Idempotency can make retries safer but does not eliminate attempt budgets or dependency overload. Reconciliation improves correctness but adds latency and requires an authoritative state. Human escalation reduces automated risk but raises operational cost.

Experienced bridge

For an experienced operator, answer five questions:

1. Where did failure occur relative to the side effect?
2. What evidence proves side effect absent, applied, or still unknown?
3. Is another attempt inside budget and safe under dependency health?
4. If the prior effect is ambiguous, what authoritative state resolves it?
5. Which evidence supports the final disposition rather than merely the last exception?

If any answer is unknown, automatic success is too strong.

Guided and independent practice

Inspect derived_text/tables/AGN-006-T1.csv, then run:

python -B lab/AGN-006-LAB1/learner/run_checks.py

For independent practice, create one new incident that fails after a side-effecting tool request and another that fails before all tools. Keep decisions different when evidence differs.

Verification evidence

Passing evidence is a reproducible ledger, not a screenshot of an exception. The lab must reject ambiguous side effects labeled safe retry, missing idempotency on side-effecting replay, exhausted attempt budget, degraded dependency without backoff/escalation, reconciliation without authority, and success with unresolved state.

No hidden model reasoning is needed to reproduce the result.

Assessment and transfer

Assessment covers boundary retrieval, baseline application, mutation diagnosis and changed-context transfer. In transfer, max-turn exhaustion occurs before any tool invocation while a dependency is degraded. Side-effect ambiguity from the baseline disappears, but dependency-health and backoff evidence now dominate. Recompute the disposition from the new failure phase.

Copying the baseline recovery decision fails transfer.

References and freshness

• OpenAI Agents SDK — Running agents / Exceptions / Tool timeout behavior: current runtime failure mechanisms only.
• AWS Builders Library — Timeouts, retries and backoff with jitter: bounded retry, overload, jitter and idempotency guidance.
• Frozen Lesson Bible lesson://AGN-006: local MAKNOW authority for the failure taxonomy, attempt budget, reconciliation and escalation design.

Recheck current SDK semantics by 2026-10-18 or earlier after a material change. Synthetic lab fixtures do not perform live retries or external actions.