
Your recurring job asks a service to create a report. The request times out, so the job tries again. Now there are two reports. The first request may have completed even though its response never reached the caller.
A timeout tells you that an expected result was not observed within a limit. It does not establish that the receiver did nothing. This guide separates retry limits from duplicate prevention, using seven small Python fixtures you can inspect and rerun.
The practical question is established, rather than invented for this article: a 2019 Python question asks how to rerun a command after a timeout. That historical question concerns retry control, not a demonstrated duplicate-write incident. The effect-versus-response distinction below comes from the AWS Builders' Library and our own synthetic experiment.
Before retrying, classify the outcome
- Known not applied: you have trustworthy evidence that this attempt never reached the effect. A retry may be appropriate if the cause is transient.
- Applied and acknowledged: retain the result and stop. A later scheduler invocation should not accidentally recreate the same logical operation.
- Outcome unknown: the reply is missing, but the effect may exist. Reconcile by operation ID, or retry under a verified idempotency contract. Do not silently turn this state into “nothing happened.”
- Invalid or conflicting intent: correct the request or ask for review. Repeating an unchanged invalid request is not recovery.
Even a successful cancellation or process termination does not undo an earlier write. A lock from our flock and timeout guide prevents a particular overlap; it does not stop the next run from repeating a completed effect.
What the seven cases actually produced
On November 6, 2026, we ran the downloadable script with Python 3.12.3 on Linux in a new directory. Each case uses a separate tiny JSON state file. A controlled exception is injected either before any effect or after a snapshot has been committed but before returning its response. No actual HTTP request, elapsed timeout or dropped network packet is involved. This makes the order of events explicit.
| Case | Attempts | Effects | Final outcome |
|---|---|---|---|
| Fault before effect; retry | 2 | 1 | Success |
| Reply lost; retry without a key | 2 | 2 | Success, duplicated |
| Reply lost; reuse the same key | 2 | 1 | Saved result replayed |
| Reply lost; change key on retry | 2 | 2 | Success, duplicated |
| Same key; change the payload | 2 | 1 | Conflict; stop |
| Two pre-effect faults; then success | 3 | 1 | Success |
| Three pre-effect faults | 3 | 0 | Attempt limit; stop |
“Success” in the second row is a caller result, not evidence that the business effect occurred only once. Both the naive case and changed-key case contain two committed effects. In the stable-key case the event sequence is:
attempt 1: committed effect 1
attempt 1: injected timeout after commit
attempt 2: replayed effect 1
attempt 2: success
The result file includes every attempt, commit/replay event, final state and expected outcome. All seven cases passed their expectations. A pass includes deliberately producing a duplicate in the unsafe cases; it does not mean every strategy is safe.
Choose a key for the operation, not for each attempt
An idempotency key expresses “these attempts belong to the same intended operation.” Create or persist it before the first attempt and reuse it after an ambiguous failure. Give a genuinely new operation a new key. The same key should be scoped to the correct caller and operation type, with a collision-resistant format appropriate to the receiving system.
Our fixture uses obvious labels such as run-001 to make the output readable. They are not a global identifier scheme. The fixture hashes a canonical JSON payload and rejects a reused key with different parameters. Hashing the payload is a consistency check, not a substitute for identifying intent: two intentional reports can have identical inputs.
Adding a header called Idempotency-Key to an arbitrary endpoint cannot create this guarantee. The receiver must implement it. Stripe's documentation, for example, describes a real provider contract: API v1 saves the status and body after endpoint execution begins, including 500 responses, compares retry parameters, and may prune keys once they are at least 24 hours old. Reusing a pruned key creates a new request. API v2 has different retry behavior. Those are Stripe-specific rules, not properties of our fixture or every API.
The effect and the receipt must agree
If the system records the key first and then crashes before writing the report, a retry may falsely look complete. If it writes the report first and crashes before recording the key, a retry may create a duplicate. The AWS article explains why recording the token and performing mutations need an all-or-nothing transaction within the relevant boundary.
For teaching, our single caller writes the effect and key record into one JSON snapshot and replaces the case's state file. It then reloads that file on the next attempt. There is no production database, locking, fsync-based durability protocol or restart test. This is not a concurrent or crash-safe storage recipe. It demonstrates the ordering and replay contract only.
Where an effect crosses systems, such as recording a row and sending an email, one local transaction does not automatically cover both. Use the recipient's supported deduplication/reconciliation mechanism and test the failure boundary. Do not claim distributed “exactly once” from this example.
Keep retries bounded even when duplicates are prevented
Duplicate prevention and retry budgets solve different problems. The last case stops at three total attempts and records zero effects. It does not loop until success or call a paid provider. Three attempts is a fixture setting, not a universal recommendation; “three retries” would normally mean four total attempts, so name the budget explicitly.
- Choose which failures are eligible using the receiver's contract. An ambiguous timeout needs different handling from a known parameter conflict.
- Set both a maximum attempt count and an overall deadline. Request timeouts, waiting and cleanup all consume the time budget. Our drill tests the attempt count only.
- Use backoff, jitter and server-directed waiting where appropriate to avoid adding pressure during an outage. We did not benchmark any delay policy here.
- When the budget ends, preserve the operation key, attempts and unresolved outcome for reconciliation. Exhaustion alone does not prove that no effect occurred; our final case has zero only because every injected fault precedes the write.
RFC 9110 section 9.2.2 defines idempotence in terms of the intended server effect, not identical response bytes. It advises against automatically retrying non-idempotent requests without knowing the semantics are idempotent or knowing the original was not applied. A method name alone does not verify an application's implementation.
Run the isolated drill
python3 retry_effect_demo.py --out ./retry-drill-output
Use python instead of python3 where that is your Python 3 command. Standard-library modules only; there is nothing to install. The script creates a new subdirectory and prints its result-file path. It never opens existing application files, sends messages or makes network requests. Inspect each case's attempts, events and final_state rather than relying only on the overall passed value.
Not tested: real request timeout behavior, concurrent workers, process restart, power loss, storage durability, provider billing, key expiry, delayed requests or distributed delivery. Before applying this pattern to a real job, test those boundaries relevant to its receiver. Continue with the small automation checklist to record success, coverage and unfinished work separately.
Sources and scope
- AWS: Making retries safe with idempotent APIs — intent identifiers, equivalent retry results, atomicity and late requests.
- RFC 9110 §9.2.2 — HTTP idempotent semantics and automatic retry limits.
- Stripe: Idempotent requests — a provider-specific contract; no Stripe request was made in this experiment.
- Historical Python timeout/retry question — historical question about retry control, verified through the public Stack Exchange API.
Sources checked November 6, 2026. The downloadable result is our own sequential synthetic measurement. The source documents explain general or provider-specific behavior; they do not validate this fixture for production.