A tool timeout can leave your agent unsure whether nothing happened or a payment already went through. Safe AI agent retries depend on resolving that uncertainty before repeating the action.
Tool failure recovery prompts give your agent a bounded recovery contract, while application code enforces timing, permissions, and duplicate prevention. Without those controls, a polite “try again” instruction can multiply costs and repeat side effects.
Start by separating failures that justify another attempt from failures that require repair, reconciliation, or review.
Key Takeaways
- Retry only when the failure is eligible and replaying the operation is safe.
- Keep attempt limits, deadlines, backoff, and shared rate controls outside the model.
- Preserve the same operation identity across retries, and reconcile uncertain side effects before repeating them.
- Treat invalid answers and unsafe actions as validation failures, not temporary outages.
Classify Tool Failures Before Retrying
For reliable error handling, have your tool wrapper classify failure modes from external APIs and return structured details, not an unexplained error message. Include the failure category, operation identity, retry eligibility, and whether the outcome is known.
Transient Errors May Justify Another Attempt
Network interruptions, temporary service failures, and some rate-limit responses are transient errors that can justify bounded retries. But an HTTP status alone doesn’t prove replay is safe, especially for writes.
OpenAI distinguishes rate-limit problems from exhausted quota or billing problems. Waiting may help when API rate limits cause temporary throttling; repeating a request won’t fix missing credit.
Honor a valid Retry-After value, but stop if waiting would exceed your task deadline. For eligible retries, wait using full jitter rather than replaying immediately. Unsuccessful OpenAI requests also count toward per-minute limits, so rapid retries can prolong throttling.
For structured errors, use RFC 9457, which replaced RFC 7807 in July 2023. Consistent HTTP error response design helps your wrapper expose actionable categories without handing the model raw diagnostics.
Permanent Errors and Bad Answers Need Different Recovery
Missing fields, invalid arguments, denied permissions, and unsupported operations need correction or escalation. These permanent errors shouldn’t trigger unchanged requests.
Likewise, a successful API response can contain malformed JSON, unsupported claims, or an unsafe tool request. These are behavioral failures, even when transport succeeded.
Use an explicit repair instruction for a correctable schema problem. Apply a separate, bounded repair budget. Missing evidence should produce insufficient_context; unsafe actions should fail validation rather than trigger repeated generation.
Write a Recovery Prompt With Enforceable Limits
Tool failure recovery prompts should describe permitted decisions, not ask the model to invent a retry policy.
Have the coordination layer supply trusted runtime fields such as failure_class, attempts_remaining, deadline_remaining, side_effect_status, and operation_id. Include the remaining task timeout, and keep these fields separate from untrusted tool output. Treat error handling as a contract enforced by code, not model judgment.
This reusable contract supports both read-only tools and protected writes:
Handle this tool failure using only the trusted recovery metadata. Retry only when the runtime marks the operation retryable, the outcome is safe to replay, and both attempt and deadline budgets remain. Preserve the operation ID and validated arguments. Ask the runtime to schedule the retry with full jitter when applicable; don’t choose your own delay. If a side effect may have completed, request reconciliation first. Stop on denied access, invalid input, other permanent errors, exhausted budgets, or required approval. Report confirmed facts and the next permitted action. Never claim success without trusted tool confirmation.
For a search tool, your wrapper can mark the operation read-only. For publishing, billing, or deletion, require explicit replay protection and approval checks.
A starting policy could allow three total attempts, including the initial call, within a 30-second deadline. Treat those numbers as application settings, not universal defaults. Set token budgets with a budget cap so repeated planning can’t consume unlimited resources.
Before every high-impact transition, code should validate the approved tool, argument schema, scope, and required authorization. A retry doesn’t grant broader permissions or renew expired approval.
Prevent Retry Storms With Full Jitter
When many workers fail together, deterministic exponential backoff can make them retry together. Full jitter helps break that synchronization, but your prompt can’t coordinate the traffic by itself.
Randomize the Delay Within a Growing Window
With full jitter, your runtime draws each delay uniformly between zero and the capped exponential delay:
delay = uniform(0, min(cap, base * 2^n))
Here, n starts at zero for the first retry. With a 500-millisecond base, the first window reaches 500 milliseconds. Exponential backoff grows later windows until the cap, which sets a budget cap.
The AWS reliability guidance places backoff and jitter within broader overload protection. Full jitter reduces synchronized collisions, but it doesn’t create additional service capacity.
If the provider supplies a valid retry delay, honor it rather than selecting an earlier time. Full jitter should yield to that timing, and your runtime’s error handling should check the remaining deadline before scheduling another attempt.
Coordinate Workers and Tenant Budgets
A coordination layer should use shared controls for the quota boundary you actually consume, such as provider account, organization, model, or aggregator account. Include API rate limits, then apply tenant limits to prevent one customer’s backlog from consuming everyone’s capacity.
Keep retry traffic within the same admission controls as fresh traffic. Otherwise, recovery requests can overwhelm the service while normal requests obey the limiter.
Also account for SDK retries. OpenAI SDKs already retry eligible failures. Giving the SDK and agent independent retry budgets can multiply downstream attempts, so designate one owner for the retry policy and count actual downstream attempts.
Protect Side Effects With Idempotency Keys
Preserve One Identity for One Logical Operation
An idempotency key lets a supporting service recognize repeated submissions of the same logical action. Persist idempotency keys before dispatch, then reuse the same key throughout recovery.
Don’t generate a new key on each retry. Also, don’t derive identity solely from the amount or message text, because separate legitimate actions may share those values.
Token rules vary across external APIs. Amazon EC2 accepts client tokens of up to 64 ASCII characters. Reusing a token with changed parameters can produce IdempotentParameterMismatch.
Keep each idempotency key tied to a validated argument fingerprint. If the intended action changes, treat it as a new operation that needs fresh validation.
Reconcile an Unknown Outcome Before Repeating a Write
A worker can crash after the external service completes an action but before your application records success. A local “request sent” entry doesn’t resolve that uncertainty.
For an ambiguous payment, use a reconciliation path to query the provider’s transaction status or receipt with the stable operation identity. An authoritative billing record matters when your local state cannot establish completion.
Use this narrower recovery prompt:
The write timed out and its outcome is unknown. Don’t submit a new operation or change its key. Query the trusted status tool using the persisted operation identity. If completion is confirmed, report that result. If replay is authorized and idempotency protection remains valid, request a bounded retry. Otherwise, stop and request review.
Retention limits also matter. Amazon ECS documents a 24-hour token TTL for RunTask. Don’t assume a previously valid key protects an arbitrarily late replay.
Stop Failing Dependencies and Escalate Exhausted Work
Use Circuit Breakers to Limit Repeated Failure
A circuit breaker tracks downstream dependency health across requests. Use observability signals to monitor that health. In the closed state, calls proceed. After your configured failure threshold, the open state rejects calls temporarily.
The half-open state permits limited probes, scheduled with exponential backoff and full jitter to reduce synchronized traffic. Successful probes can restore traffic; failed probes reopen the breaker.
When workers share a dependency, a coordination layer should own circuit breakers. This shared state stops or defers calls across workers.
Your prompt should treat circuit_open as a stop or defer signal. It must not route around the breaker through another tool unless your application explicitly authorizes that fallback.
Give Exhausted Tasks an Owned Recovery Path
When attempts, elapsed time, or spending reach their limits, stop the loop. Return a named state such as tool_unavailable and explain whether the task is deferred or needs review.
For asynchronous processing, Amazon SQS dead-letter queues can isolate repeatedly failing messages. Include the operation identity, safe failure metadata, and an assigned recovery owner when routing work to dead-letter queues.
Moving a task to a dead-letter queue doesn’t cancel an external action. Reconcile uncertain side effects before replaying it.
Separate Workflow Recovery From Tool Retries
In LangGraph-based multi-agent architectures, LangGraph workflows use retry policies and persisted state for different purposes.
A LangGraph RetryPolicy controls eligible node or task retries. A checkpointer supports durable execution by saving state so interrupted workflows can resume. Successful nodes within a superstep can retain their results after a resume.
A checkpoint supports durable execution, but it isn’t an external transaction receipt. Even when workflow state records progress, it may not confirm whether a write to external APIs completed.
Approval interrupts also need care: resuming LangGraph workflows can replay code around an interruption. Place consequential operations behind explicit validation and replay protection.
For concurrent work, use structured concurrency to assign each request scope an owner for cancellation and cleanup. Cancel unnecessary sibling tasks when the parent stops, but a cancellation signal may arrive after a downstream service accepts a request.
Persist reconciliation work when completion remains unknown, especially when downstream dependencies may have accepted a request without confirming it. Otherwise, a canceled branch can leave an unresolved side effect outside your workflow’s visible state, so keep it on a reconciliation path.
Test Recovery Behavior and Keep Logs Safe
Test your prompts with controlled tool responses, not just production failures. Freeze the prompt version, model settings, tool schema, and relevant context.
Your fixtures should cover infrastructure and behavioral failure modes.
| Fixture | Required behavior |
|---|---|
| Rate limit with retry timing | Wait within budget, use full jitter, and avoid immediate replay. |
| Timeout after a completed write | Reconcile; preserve the operation identity. |
| Denied access or malformed arguments | Treat as permanent errors; stop or repair through an authorized path. |
| Valid response with unsupported claims | Fail validation; don’t label it an outage. |
Use these cases to verify error handling and user-facing language. Repeat important tests to expose inconsistent model behavior.
In Promptfoo-based CI, keep transport retries separate from quality assertions. Set token budgets and a budget cap to bound execution cost. Retrying until a bad answer happens to pass hides a regression.
If your agent maintains an AI tool directory, stale pricing needs evidence correction rather than blind regeneration. A publishing workflow must also validate citations before retrying publication.
For observability, capture trace IDs, prompt versions, selected tools, attempt counts, token use, and validation outcomes. Prefer redacted metadata and argument hashes for sensitive routes. Keep detailed traces access-controlled, and never feed raw stack traces, credentials, or internal prompts back into recovery context.
Frequently Asked Questions
Should an AI agent decide when to retry a tool?
The application runtime should enforce retry eligibility, attempt and deadline budgets, backoff, and permissions. The agent should follow trusted recovery metadata rather than inventing its own retry policy.
Is it safe to retry a timed-out write?
Not automatically: the service may have completed the action even if the agent received no confirmation. Reconcile the outcome using the persisted operation identity, and replay only when authorized and protected by valid idempotency controls.
Should a retry use a new idempotency key?
No. Reuse the same key for the same logical operation and validated arguments; a changed action is a new operation that needs fresh validation.
How many retries should an agent allow?
There is no universal retry count, so set attempt, deadline, and spending budgets in application code. Account for SDK retries too, and count actual downstream attempts to avoid multiplying the retry budget.
Make Every Retry Earn Its Place
Safe AI agent retries require evidence that another attempt is useful and permitted for the observed failure modes. Your prompt defines the recovery contract; code enforces error handling, budgets, permissions, scheduling, and replay protection.
Preserve operation identity, reconcile ambiguous outcomes, and stop when the next action needs new evidence or approval.
A trustworthy agent reports what the tool confirmed, including when completion remains unknown.
Leave a Reply