Idempotency and retries: building automations that don't break
TL;DR
- Retries are not optional. Networks drop, services hiccup, and any automation that talks to another system will eventually need to try again.
- Naive retries are dangerous. The same request sent twice can charge a card twice or send the same email twice unless the operation is built to be safely repeatable.
- Idempotency is the fix: give each operation a stable key, record what you have already done, and check that record before acting again.
Why your automations need retries in the first place
Every automation you build that crosses a network boundary is making a bet that the other side will answer. Most of the time it does. But over thousands of runs, you will hit the cases that break naive code: a payment gateway that times out under load, a CRM API that returns a 503 for ninety seconds during a deploy, a webhook that arrives twice because the sender did not get your acknowledgement fast enough.
None of these are exotic. They are the normal weather of distributed systems. We have watched a perfectly correct integration sit in production for months, then fall over on a Monday morning simply because a downstream service was slow that day. The code was fine. The assumption that calls always succeed was not.
So you add retries. If a call fails, try it again. That instinct is right, and a system without retries is brittle in a way that shows up at the worst possible time. The trouble is that retries done carelessly trade one failure mode for a worse one, and the worse one is harder to spot because it looks like success.
- Transient network errors: dropped connections, DNS blips, TLS handshakes that stall.
- Service-side hiccups: rate limits, brief outages during deploys, overloaded databases.
- Timeouts where you never learn whether the request actually landed.
- Duplicate deliveries from upstream senders that retry their own webhooks.
How naive retries cause double charges and duplicate emails
Here is the concrete failure we have cleaned up more than once. An automation calls a payment API to charge a customer. The request reaches the gateway, the charge succeeds, and then the response on the way back times out. Your code never sees a success, so it does what you told it to do: it retries. The gateway, with no way to know this is the same charge, processes a second one.
Now the customer has paid twice. Your logs show one failure and one success, so nothing looks wrong from where you sit. The first signal you get is a support ticket and a chargeback. The same shape of bug sends two welcome emails, creates two CRM records for one lead, or ships two orders for one purchase.
The root cause is not the retry. The retry was the right move. The root cause is that the operation was not safe to repeat. Sending the same request twice produced two real-world effects instead of one. Fixing that is the whole point of idempotency, and it is the difference between an automation that survives flaky networks and one that quietly corrupts your data.
- Double charges when a payment succeeds but the response is lost.
- Duplicate emails or SMS when a send is retried after a timeout.
- Two records for one entity when a create call runs twice.
- Duplicate fulfilment when an order webhook is delivered more than once.
What idempotency means in plain terms
Idempotency is a heavy word for a simple idea: doing the same thing twice has the same effect as doing it once. If an operation is idempotent, you can retry it as many times as you like and the end state is identical to running it a single time. That property is what makes retries safe.
Some operations are naturally idempotent. Setting a customer's status to active is idempotent, because setting it again changes nothing. Reading data is idempotent. Deleting a record by id is usually idempotent, since deleting an already-deleted record is a no-op. The dangerous operations are the ones that add or create: charge a card, append a row, send a message. Each repeat does real new work.
The job, then, is to take those create-style operations and make them behave like the idempotent ones. You do not change what the operation does. You change how the system remembers whether it has already happened, so the second attempt becomes a safe no-op that returns the original result instead of doing the work again.
- Naturally idempotent: reads, setting a value to a fixed state, delete-by-id.
- Not idempotent by default: charges, message sends, row inserts, counters.
- The goal: make create-style work repeatable without producing extra effects.
- The test: run it twice and confirm the end state matches running it once.
Idempotency keys and dedupe stores
The standard tool for this is an idempotency key. Before you perform an operation, you generate a stable identifier that represents this specific intent: charge order 4471, send the welcome email to lead 9920. The key has to be the same on a retry, so derive it from the work itself rather than generating a fresh random value each attempt. A natural business id plus the action is usually enough.
You then need somewhere to record that key: a dedupe store. Before acting, you check whether the key already exists. If it does, you skip the work and return the stored result. If it does not, you do the work, then write the key and its outcome. Most good payment and messaging APIs accept an idempotency key header directly and run this check for you, which is the easiest win available. When the downstream service does not support keys, you run the check yourself in your own store.
A small but important detail: store the result alongside the key, not just the fact that you acted. When a retry hits an existing key, the caller still expects an answer. Returning the original charge id or message id keeps the rest of your automation working as if the first call had simply succeeded, which is exactly the illusion you want.
- Derive keys from stable business facts, not from a new random value per attempt.
- Use the provider's idempotency-key header when it offers one.
- Keep a dedupe store (a table, a key-value store) keyed by idempotency key.
- Record the outcome with the key so retries can return the original result.
- Set a sensible retention window so the store does not grow without limit.
Designing operations to be safely repeatable
Idempotency keys are the mechanism, but the deeper habit is designing each step so repeating it cannot hurt. The cleanest pattern is check-then-act inside a single guarded path: look for an existing record by the idempotency key, and only do the work if you do not find one. Where your database supports it, an insert that ignores or rejects duplicate keys gives you this guarantee at the storage layer, which is far safer than checking in application code where two retries can race.
Order matters too. Write the dedupe record and do the external action in a way where a crash in the middle does not leave you in an ambiguous state. A common approach is to reserve the key first in a pending state, perform the action, then mark it complete. If a retry finds a pending record, it can check the downstream system for the real outcome rather than blindly redoing the work.
If you are building or auditing automations and want a second set of eyes on these failure paths before they reach production, that is exactly the kind of review our team does day to day. The patterns are not hard, but the edge cases are easy to miss when you are heads-down shipping.
- Prefer check-then-act guarded by a unique constraint over in-memory checks.
- Use insert-if-not-exists at the database level to win the race for you.
- Reserve the key in a pending state before the external call, complete it after.
- On a retry of a pending key, reconcile with the downstream system, do not re-run blindly.
Transient versus permanent errors, and how to retry each
Not every failure deserves a retry. The first decision your error handling has to make is whether an error is transient or permanent. A transient error is one that might succeed if you try again: a timeout, a 429 rate limit, a 503, a connection reset. A permanent error will fail every time no matter how many attempts you make: a 400 bad request, a 401 with bad credentials, a 422 validation failure. Retrying a permanent error just wastes time and hammers a service that is telling you to stop.
So classify before you retry. Treat most 5xx responses and network-level failures as retryable. Treat 4xx responses, with the exception of 429, as permanent and route them straight to your failure handling. The exact list depends on the API, so read its docs rather than guessing, but the principle holds everywhere: retry the things that can recover, surface the things that cannot.
Getting this split right is half the battle. We have seen automations that retried a malformed request a hundred times because the handler treated every non-200 the same. The fix was not more retries. It was teaching the code to tell the two kinds of failure apart and act differently.
- Retryable: timeouts, connection resets, 429 rate limits, most 5xx errors.
- Permanent: 400, 401, 403, 422 and similar validation or auth failures.
- Read the specific API docs; classification differs between providers.
- Send permanent errors straight to alerting, not into the retry loop.
Backoff, jitter, caps, and the retry-after-timeout trap
Once you know an error is retryable, how you retry matters. Retrying instantly and forever is its own outage: a downstream service comes back to life and gets slammed by every client retrying in lockstep, knocking it over again. The standard answer is exponential backoff. Wait a short delay, then double it on each attempt, so one second, then two, then four, giving the struggling service room to recover.
Backoff alone is not enough, because if every client uses the same delays they retry at the same moments and create waves. Add jitter, a small random amount on each wait, to spread retries out. And cap the whole thing: a maximum delay so you do not wait minutes, and a maximum number of attempts so a hopeless request does not loop forever. Three to five attempts covers the vast majority of transient failures.
Now the trap that ties this back to idempotency. The most dangerous failure is the timeout, because a timeout does not tell you whether the work happened. The request may have landed and succeeded while only the response was lost. If you retry that without an idempotency key, you risk the double charge. This is precisely why backoff and idempotency are a pair, not two separate topics. Backoff decides when to retry; idempotency makes that retry safe even when you cannot tell what the first attempt did.
- Exponential backoff: grow the wait on each attempt to let the service recover.
- Jitter: add randomness so clients do not retry in synchronised waves.
- Caps: a maximum delay and a maximum attempt count, often three to five.
- Never retry after a timeout without an idempotency key protecting the call.
Dead-letter queues, alerting, and a practical checklist
Retries buy you resilience, but some requests will exhaust every attempt. The wrong move is to drop them silently, which turns a recoverable hiccup into lost data nobody notices for a week. The right move is a dead-letter queue: when a message fails past its retry budget, move it somewhere durable for inspection instead of throwing it away. A dead-letter queue turns a permanent failure into a triageable backlog you can replay once the underlying problem is fixed.
Pair that with alerting that is loud enough to act on but quiet enough to trust. Alert when items land in the dead-letter queue, when retry rates spike, or when a single key keeps reappearing. The aim is to learn about trouble from your own monitoring, not from a customer. A dead-letter queue with no alert is just a place where data goes to be forgotten.
Start here. If you are hardening an automation today, walk the checklist below in order. Classify your errors, make the risky operations idempotent, add backoff with jitter and caps, then catch whatever still fails in a dead-letter queue with an alert on top. That sequence takes a fragile script and makes it something you can leave running without holding your breath.
- Add idempotency keys to every create-style operation that can be retried.
- Classify errors as transient or permanent and handle each differently.
- Use exponential backoff with jitter and a hard cap on attempts.
- Treat timeouts as unknown outcomes, protected by idempotency, never blind retries.
- Route exhausted retries to a dead-letter queue with alerting and a replay path.
Want this built for your business?
We map the highest-leverage place to start and ship a first live system within two weeks.
Book a strategy callCommon questions
What is idempotency in simple terms?
It means doing the same operation more than once has the same effect as doing it once. If an operation is idempotent, you can safely retry it without creating duplicate charges, emails, or records, because repeat attempts do not produce extra real-world effects.
Why do retries cause double charges?
Usually because of a timeout. The payment reaches the gateway and succeeds, but the response is lost on the way back, so your code never sees success and tries again. Without an idempotency key, the gateway treats the second attempt as a brand new charge.
What is an idempotency key?
A stable identifier that represents one specific intent, such as charging order 4471. You send the same key on every retry, and the service or your dedupe store uses it to recognise a repeat and return the original result instead of doing the work twice.
Which errors should I retry and which should I not?
Retry transient errors that might recover: timeouts, connection resets, 429 rate limits, and most 5xx responses. Do not retry permanent errors like 400, 401, or 422, since they will fail every time. Send those straight to your failure handling and alerting.
What is a dead-letter queue and do I need one?
It is a durable place to park messages that have failed past their retry budget, so they are not lost. You need one for any automation where dropped work matters. Pair it with alerting so you find out about failures from monitoring rather than from a customer.
It means doing the same operation more than once has the same effect as doing it once. If an operation is idempotent, you can safely retry it without creating duplicate charges, emails, or records, because repeat attempts do not produce extra real-world effects.
Ask AI about X18 Global
“What does X18 Global (x18global.com) do for enterprise AI and automation - and can you summarise their guide "Idempotency and retries: building automations that don't break"?”