The customer got two refunds. The agent swears it sent one.

This is one of the most important concepts in the course because checkpointing does not solve it.

First define a side effect

A side effect is an operation that changes something outside the workflow’s internal state.

Examples:

send email
issue refund
publish report
create Jira ticket
write CRM record
deploy code
change permissions

Fetching a page is usually a read.

Publishing the final report is a side effect.

The ambiguous-failure timeline

Agent                     Report API
  │                           │
  │ POST /reports             │
  ├──────────────────────────►│
  │                           │ create report #1001
  │                           │ commit
  │                           │
  │             response      │
  │◄────────────── X ─────────┤
  │
  │ timeout

The agent knows:

I did not receive a success response.

It does not know:

The report was not created.

That distinction is the core problem.

Possible external reality after an error

A write may be:

not performed
performed completely
performed partially

The client may not know which one is true.

Why generic retries are dangerous

Naive code:

for attempt in range(3):
    try:
        return publish_report(report)
    except TimeoutError:
        continue

If the first attempt committed and only its response was lost:

attempt 1 → report #1001
attempt 2 → report #1002

The retry turned uncertainty into duplication.

Idempotency

An operation is idempotent when repeating the same logical operation does not produce an additional unintended business effect.

Logical operation:

Publish the final report for run_123.

Stable operation identity:

idempotency_key = "run_123:publish_final"

The same key must be reused across retries.

Bad:

f"run_123:attempt_{attempt}"

That creates a new operation identity on every attempt.

Enforce identity where you own the target

CREATE TABLE published_reports (
    report_id BIGSERIAL PRIMARY KEY,
    run_id TEXT NOT NULL UNIQUE,
    body JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Now the database itself prevents two final reports for the same run.

Request hash

A deeper production detail: the same idempotency key should not silently accept a different request.

Store:

effect_key
request_hash
external_id
status

Behavior:

same key + same request
    → return previous result

same key + different request
    → conflict; investigate

This prevents accidental reuse of one key for two meanings.

Side-effect journal

Create an application-owned record before/around an important external action:

CREATE TABLE side_effects (
    effect_key TEXT PRIMARY KEY,
    run_id TEXT NOT NULL,
    effect_type TEXT NOT NULL,
    request_hash TEXT NOT NULL,
    status TEXT NOT NULL,
    external_id TEXT,
    last_error TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Status might be:

planned
in_flight
succeeded
needs_reconciliation
failed

This gives the system something durable to inspect after a crash.

Reconciliation

Reconciliation means querying external reality after an ambiguous result before deciding whether another write is safe.

publish timed out
  ↓
status = needs_reconciliation
  ↓
query report service by stable key/reference
  ↓
report exists?
  ├─ yes → record external ID, mark succeeded
  └─ no  → retry same logical operation if safe

If an external service provides neither idempotency nor a way to query by a stable business reference, autonomous writes to it are much harder to make reliable.

That is an architectural constraint, not a prompt problem.

Exactly-once language

Do not casually claim “exactly once” across arbitrary distributed systems.

A practical design is often:

at-least-once attempts
+ stable operation identity
+ deduplication/idempotent target
+ reconciliation
= one intended business effect

Be explicit about the mechanism.

Transaction boundary

A database transaction can make changes atomic inside that database.

It does not magically include a remote SaaS API.

If you must update local state and later notify another system, introduce the outbox pattern as an advanced production extension:

local DB transaction
  ├─ update business state
  └─ insert outbox event
commit

outbox dispatcher
  ↓
external API

The dispatcher may still deliver more than once, so the external effect still needs stable identity.

attempt 1:
  create report
  drop response

attempt 2:
  return normally

SELECT COUNT(*)
FROM published_reports
WHERE run_id = 'run_123';

Side-effect risk classes

Teach this classification:

READ
search, fetch, inspect

REVERSIBLE WRITE
create a draft, create temporary branch

EXTERNAL / IRREVERSIBLE WRITE
send, publish, charge, refund

PRIVILEGED WRITE
delete, change access, production deploy

The stronger the effect, the stronger the identity, authorization, review, and audit requirements.

Check your understanding

Exit criteria