How to Make API Retries Safe With Idempotency Keys
A timeout does not tell you whether the request worked. Idempotency keys let clients retry without charging a customer twice.
On this page 6 sections
An idempotency key is a unique value a client sends with a request so the server can recognize a retry and return the original result instead of repeating the action. To use them, generate a key per operation, store it with the response, and reject reuse with different parameters.
This guide is for backend engineers who run APIs that move money, create orders, or send messages. Backtrace writes about the tradeoffs of production systems, and duplicate side effects are one of the most expensive bugs we see described in postmortems. The advice here is evergreen and applies to any HTTP API.
Why API Retries Create Duplicate Charges
A client sends a payment request and the connection drops. The client sees a timeout and knows nothing else. The server may have processed the charge, may have failed halfway, or may never have received the call.
The only safe-looking choice is to retry, and retrying is exactly how duplicates happen. A load balancer can also retry on its own, and so can a mobile app that comes back online. Each layer is trying to be helpful, and together they can fire the same request three times.
The HTTP standard already draws a line here. The IETF's RFC 9110 section on idempotent methods says PUT and DELETE are idempotent by definition, meaning repeating them has the same effect as sending them once. POST has no such guarantee, and POST is the method most create and charge endpoints use.
Consider a hypothetical API that handles one million payment requests a day. If just 0.1 percent of them time out and get retried, that is 1,000 potential duplicate charges every day. Each one becomes a support ticket, a refund, and a little less trust.
Idempotency keys solve this by making the retry safe instead of trying to prevent it. The server remembers what it did the first time. A second request with the same key gets the same answer and causes no new side effect.
What Makes an Idempotency Key Work?
A good key identifies one logical operation, not one HTTP attempt. The client creates it before the first try and reuses it on every retry of that operation. A new operation, such as a second purchase, gets a new key.
Use a random UUID version 4 or a similarly unguessable string. Do not build keys from user ID and timestamp, because two legitimate requests can collide, and a predictable key invites abuse. Most providers accept up to a few hundred characters, so length is rarely the limit.
The server needs to do more than look the key up. It should also store a fingerprint of the request, such as a hash of the body and the target path. If the same key arrives with different parameters, the server must reject it with an error. Otherwise a client bug could silently return the wrong saved result.
Stripe's documentation on idempotent requests is a useful real-world model. It saves the status code and body of the first request, including errors, and returns them for later requests with the same key. It also says keys can be removed from the system after they are at least 24 hours old.
Scope each key to the caller. A key should be unique per account or API credential, not global. Otherwise one customer's key can collide with another's, and a malicious client could probe for keys that belong to someone else. Credentials need the same care, a topic we covered in our guide to API key rotation.
Where Should You Store Idempotency Keys?
Store them in the same database transaction as the work they protect, in a table with a unique constraint on the key. That way the key and the side effect commit together or not at all. A cache alone can lose entries and let a retry run twice.
The unique constraint is what makes this safe under concurrency. Two retries can arrive at the same moment on different servers. Both try to insert the key, one wins, and the other receives a conflict error that it can turn into a "request in progress" response.
A minimal table needs a handful of columns. The list below names them and what each one holds.
account_idandkey, together forming the unique constraint.request_hash, the fingerprint used to detect a mismatched retry.status, which is started, completed, or failed.response_codeandresponse_body, saved from the first attempt.created_at, used to expire old rows.
Adding this table to a busy database is a schema change like any other. Create it before the code that uses it ships, and use the approach described in our walkthrough of zero-downtime schema migrations. A new table is low risk, but a new index on a large existing table is not.
If the side effect lives outside your database, such as a call to a card processor, the transaction cannot span both. In that case, pass your own key downstream to the processor as well. Then a retry of your request becomes a retry of theirs, and they dedupe it for you.
How to Add Idempotency Keys in Five Steps
Roll the feature out on one endpoint first, ideally the one with the costliest duplicate. Follow these steps in order.
- Pick the endpoint, and require or accept an
Idempotency-Keyheader on its POST route. - Create the key table with a unique constraint on account and key.
- On each request, insert the key with status started, and return a conflict if the insert fails.
- Run the operation, then save the response and mark the key completed in the same transaction.
- On a repeat request, compare the request hash, then replay the saved response with a header that marks it as a replay.
Decide whether the header is required or optional. Required keys are safer for payment endpoints because they prevent a careless client from skipping protection. Optional keys are easier to adopt for existing clients, and you can tighten the rule later.
The table below summarizes how the server should respond in each situation. Treat it as the contract your documentation promises.
| Situation | Server action | Typical response |
|---|---|---|
| New key | Process and save the result | Normal success or error |
| Same key, same request, finished | Replay the saved result | Original status and body |
| Same key, same request, still running | Do nothing new | 409 Conflict, retry later |
| Same key, different request | Reject | 422 or 400 with a clear message |
| Key past expiry | Treat as new | Normal processing |
Teams often ask whether to save failed results. For client errors such as validation failures, saving is fine, since the same input will fail again. For server errors, think harder. Some providers replay even a 500, while others release the key so the client can try again after a fix.
Roll the change out behind a flag so you can turn it off quickly. The practices in our feature flags guide apply directly. Enable it for internal traffic, then a small share of clients, and watch the conflict rate before widening.
Edge Cases That Trip Teams Up
The first edge case is a crash between the side effect and the saved response. If you write both in one transaction, this cannot happen. If you cannot, the operation needs its own recovery job that checks started keys older than a few minutes and finishes or cancels them.
The second is key lifetime. Expire keys on a published schedule, such as 24 hours or seven days, and say so in your docs. A client that retries after the window will create a duplicate, and that is a documented limit, not a bug.
The third is clock and ordering trouble in distributed setups. Do not rely on timestamps to decide which request came first. Let the database constraint decide, since it is the single point that sees every attempt.
One honest limit remains. Idempotency keys protect against duplicate delivery, not against a client that sends two different keys for the same intent. Pair them with sensible client behavior, such as disabling a submit button after one click.
Test the contract before clients depend on it. Write automated tests that send the same request twice, send it twice at once, and send it with a changed body. Each test should confirm that only one side effect happens and that the error messages are clear enough for a client developer to act on.
Conclusion
Safe retries come from a simple contract. The client names the operation, the server remembers the outcome, and a repeat returns the same answer. A unique constraint and one transaction do most of the work.
Start with your most expensive endpoint this sprint. Add the key table, ship the replay path behind a flag, and watch the conflict rate for a week. Backtrace readers who run payment or order APIs should treat this as baseline hygiene rather than an advanced feature.
