The log store is down. Should the investigation die?

“Retry three times” is not a failure strategy.

Step 1: classify the failure

The first production decision is not “how many retries?”

It is:

What kind of failure is this?

Transient infrastructure failure

Examples:

503
429
connection reset
temporary DNS problem

Often retryable.

Permanent failure

Examples:

resource does not exist
credential permanently lacks access
operation unsupported

Repeating the same call unchanged will not fix it.

User/configuration failure

Examples:

missing credential
invalid root URL
required authorization absent

The system may need human/configuration input.

Model-recoverable failure

Examples:

invalid structured output
bad tool argument
poor query

The correct path may be to expose the structured error to the planner and let it choose a corrected action.

That is different from infrastructure retry.

Ambiguous external write

Example:

publish timed out after remote commit may have happened

This routes to reconciliation, not blind retry.

Unexpected software failure

Examples:

KeyError
state invariant violation
corrupt artifact

A deterministic software bug usually fails again if retried unchanged.

Fail fast, capture evidence, and alert rather than hiding it behind repeated attempts.

Retry policy

A real retry policy includes:

RetryPolicy(
    max_attempts=4,
    initial_backoff_s=1,
    max_backoff_s=20,
    timeout_s=15,
    jitter=True,
    retryable_errors={
        RateLimitError,
        ServiceUnavailable,
        ConnectionResetError,
    },
)

Exponential backoff

Instead of:

retry now
retry now
retry now

use something like:

1 second
2 seconds
4 seconds
8 seconds

up to a configured cap.

Jitter

Jitter adds randomness to retry timing.

Why?

If 500 workers see the same outage at the same time and all retry at exactly 2, 4, and 8 seconds, they create synchronized traffic spikes.

Jitter spreads those attempts out.

Timeouts are layered

Explain separately:

connection timeout
    how long to establish a connection

read timeout
    how long to wait for response data

operation timeout
    maximum duration of one logical tool operation

run deadline
    maximum duration/age of the business run

A hung tool call should not consume the entire workflow’s time budget by accident.

Retry budget

Per-call retry policies can multiply across a run.

If 20 tool calls each retry five times, the total failure amplification is huge.

Track a run-level budget:

state["retry_budget_remaining"] = 12

Every retry consumes from it.

Model correction loop

Example invalid tool call:

{
  "tool": "fetch_page",
  "url": null
}

Possible route:

schema validation error
  ↓
structured observation returned to planner
  ↓
planner chooses corrected tool argument

Bound this loop too.

max model correction steps = 5

Partial completion

A run that cannot verify every requirement should not invent completeness.

Useful terminal result:

{
  "status": "completed_with_unknowns",
  "terminal_reason": "budget_exhausted",
  "verified": [
    "product",
    "customers",
    "security"
  ],
  "unknown": [
    "pricing"
  ]
}

Partial but honest can be the correct result.

Circuit breaker

Introduce as a system-level concept.

If a shared model provider is failing for every run, per-run retries can worsen the outage.

A circuit breaker moves through states like:

CLOSED
calls allowed
  ↓ repeated failures
OPEN
calls temporarily blocked
  ↓ cooldown
HALF-OPEN
small probe traffic
  ↓ recovery
CLOSED

The core lab need not build a production circuit breaker, but the concept should be clear.

503 twice then 200
404
401
malformed model output
30-second hang
connection reset
ambiguous publish timeout
unexpected Python exception

503                  retry with backoff
404                  mark unavailable / choose alternative
401                  configuration/human action
malformed model      bounded model correction
hang                 timeout then classify
ambiguous publish    reconcile
invariant violation  fail fast + alert

Check your understanding

Exit criteria