The Idempotency Engine: Ensuring Transactional Integrity

SchemaBridge Team · 2025-12-22 · Idempotency, Consistency, Transactions

At-least-once execution, inbound dedup, and destination-side keys: where duplicates are actually stopped in a distributed workflow, and which part is your job.

The "Double-Charge" Nightmare: Why Distributed Systems Hate Retries

In our world of flaky networks and ephemeral cloud resources, failures are not just frequent; they are a constant state of existence. Every senior engineer has experienced the "Orphaned Action" nightmare, a scenario that starts as a minor network glitch and ends as data corruption or a financial liability. It's the classic distributed systems failure scenario:

1. Request: Your orchestrator sends a "Charge" request to a payment API or a "Ship" request to a carrier.

2. Success: The third-party API successfully processes the request, charges the customer's card, or prints the shipping label.

3. Partition: A brief network glitch occurs in the return path. The response from the API—the vital confirmation—never reaches your worker machine.

4. Retry: Your worker, seeing a timeout, assumes following standard practice that the process failed. It follows its retry policy and sends the request again.

5. Duplicate: Because the API hasn't been given a unique identity for this specific intent, it processes the request again. It sees it as a new transaction. The customer is charged twice, or two shipping labels are generated for the same order.

This is not a failure of the code logic; it is a failure of Identity. Without a way to uniquely identify a specific intent across time and space, your system is gambling with your customers' data and money every time it retries a connection.

Distributed Transactions are Dead: Long Live Idempotency

In a monolithic architecture, we rely on Two-Phase Commit (2PC) or globally distributed locks to ensure consistency. These tools allow us to treat multiple operations as a single unit of truth. But in a fragmented world of SaaS APIs, serverless workers, and polyglot microservices, global transactions are a fantasy. They don't scale, they introduce massive latency, and most third-party providers don't support them. They require a "Locking" of resources that is impossible to achieve across organization boundaries.

The only viable path for distributed consistency is Idempotency. Mathematically, an operation is idempotent if it can be applied multiple times without changing the result beyond the initial application. In algebraic terms: f(x) = f(f(x)). In engineering terms, it means your system can fail and retry any number of times, and the end result will always be the correct one.

The Honest Contract: At-Least-Once, With Guards

Let's be precise about what an orchestrator can and cannot promise. SchemaBridge delivers work at-least-once. A worker can crash after the HTTP call succeeded but before it recorded the result; a message can be redelivered; a claim lease can expire and be taken over. In each of those cases, the safe behaviour is to run the step again rather than lose it — which means a step can, rarely, run twice.

What the platform does is make that "rarely" as rare as it can, at two layers:

Layer 1: Inbound Trigger Deduplication

Every trigger of an entry point gets an execution ID, and by default that ID is derived from the content: a deterministic SHA-256 over the workspace, the entry-point vertex, and the request payload. The same webhook delivery arriving twice resolves to the same execution instead of starting a second one — which is exactly what you want when a sender retries. You control the identity with three knobs:

| Knob | Where | Use when |

| :--- | :--- | :--- |

| idempotencyFields | Entry-point vertex config | You want dedup keyed on a business identifier (e.g. order.id) rather than the whole body |

| clientIdempotencyId | Query parameter on the trigger call | The caller owns the dedup key |

| forceNewExecution | Test runs and the streaming execute call only (on by default there) | Not available on production triggers — there, use idempotencyFields or a unique clientIdempotencyId per call |

{
  "type": "ENTRY_POINT",
  "config": {
    "idempotencyFields": "order.id"
  }
}

The flip side is a real gotcha: if two legitimate runs can carry byte-identical payloads ("run the nightly report with an empty body"), they will collapse into one unless you opt out. Scheduled (cron) triggers are the exception — each fire gets a fresh random ID, because a schedule is never meant to be deduplicated. And be aware that explicit dedup is a read-then-write lookup, not a database uniqueness constraint, so two truly concurrent triggers with the same key can both get through.

Layer 2: Step-Level Guards

Inside a run, a step that is already COMPLETED is not executed again, a worker must atomically claim a step (DISPATCHED → EXECUTING) before running it, terminal transitions are gated, and a claim epoch fences off a worker whose lease was taken over. These guards stop the common duplicate paths — but none of them can reach into a third-party API and undo a call that already happened.

Layer 3 Is Yours: The Destination-Side Key

That last gap is why the platform's own behaviour contract says it plainly: make the side effect idempotent at the destination, and don't rely on the orchestrator to run it exactly once.

The good news is that most serious APIs (payments, shipping, messaging) accept an idempotency key, and a Gateway step can send one derived from the run's data. A header value in SchemaBridge can be an expression over the step's input, so the key is computed from a stable business identifier — never from a timestamp or a random value:

{
  "type": "GATEWAY",
  "config": {
    "method": "POST",
    "url": "https://api.payments.example/v1/charges",
    "headers": {
      "Idempotency-Key": "= 'charge-' & payload.order.id",
      "Authorization": "Bearer {{ secrets.PAYMENTS_KEY }}"
    }
  }
}

Because the key depends only on the order, a re-run of this step — after a redelivery, an operator Retry, or an automatic overload retry — sends the same key, and the destination returns the first result instead of charging again. One caveat: Start over (a replay of the whole workflow) creates a new execution, but if your key is derived from business data rather than the execution ID, it stays stable even then. That is precisely why business identifiers make better keys than engine identifiers.

Handling Legacy APIs: The "Read-Verify-Write" Pattern

Unfortunately, many legacy systems and niche SaaS providers don't support idempotency keys. They don't have an Idempotency-Key header. For these "Un-idempotent" endpoints, you can compose a Read-Verify-Write sequence out of ordinary steps:

1. Verifier Step (Read): A Gateway step first queries the downstream system to see if the record already exists (e.g., GET /orders?external_id=123, using the same business identifier you would have used as a key).

2. Condition Branch: A conditional edge checks the response. If the order exists, the run takes the "already done" path. If not, it proceeds.

3. Action Step (Write): Only if the Verifier finds nothing does the run proceed to the actual write (POST /orders).

Know the Window

Read-Verify-Write narrows the duplicate window; it does not close it. If a worker crashes after the write but before recording success, the write step is re-run on its own — it does not go back and re-verify first. So for a destination with no idempotency support at all, also look for a natural uniqueness constraint on the destination side (an external_id it rejects on conflict), and treat a conflict response as "already done". By default a 4xx fails a Gateway step; setting clientErrorMode to complete turns a client-error response into data your next step can route on.

The "Key Management" Trap: Why DIY Idempotency Fails at Scale

Many engineering teams attempt to build an "Idempotency Table" in their primary database. This creates three critical problems that ultimately kill velocity and reliability:

1. The Write Bottleneck: Every single API call now requires a database write to record the token. Under heavy load, your idempotency table becomes a point of contention.

2. Cleanup Complexity: The Garbage Problem: Idempotency keys aren't needed forever. You need a background process or a TTL (Time-To-Live) to prune old keys. If your pruning is too aggressive, you risk double-charges for slow, retried tasks. If it's too slow, the table grows without bound.

3. The Distributed State Mismatch: What happens if the database write succeeds but the API call fails? Or what if your worker crashes after the API call but before the database can be updated to say "Finished"? You end up with a mismatch that requires manual intervention to resolve.

The way out is not a smarter table on your side; it is to let the system that performs the side effect own the dedup. A destination that honours an idempotency key has already solved storage, expiry, and the crash window for its own operation — your job is to send it a stable key.

Client-Side vs. Server-Side Token Generation

Where should the token be generated?

Client-Side (at the source of intent): The caller derives the token before the first attempt and reuses it on every retry. This is the only placement that protects against the "Return-path Failure" described at the beginning of this post.
Server-Side: The receiver generates an ID (usually a DB ID). This is useful for internal consistency but doesn't help a caller whose confirmation was lost.

In a SchemaBridge workflow, the source of intent is the run's own data. Deriving the key there, in the Gateway step's header, keeps it stable across every retry path the engine might take.

Comparison Table: Where Duplicates Are Stopped

| Duplicate source | Stopped by | Who configures it |

| :--- | :--- | :--- |

| Sender retries the same webhook | Content-derived execution ID / idempotencyFields / clientIdempotencyId | SchemaBridge, with your choice of key |

| Step redelivered inside a run | Step-level guards (completed short-circuit, atomic claim, claim epoch) | SchemaBridge |

| Step re-runs after the call already succeeded | Destination-side idempotency key | You, on the Gateway step |

| Destination has no idempotency support | Read-Verify-Write plus a destination uniqueness constraint | You |

Expert Tips: The Idempotency Checklist

1. Never Use Timestamps: Your key must be based on the data, not the time.

2. Scope Your Keys: Prefix keys by operation (charge-, ship-) so a key for "Shipping" never collides with a key for "Billing" even if they share an order ID.

3. Handle 409 Conflicts: If an API returns a 409 (Conflict) for a key or identifier it has already seen, your workflow should usually treat it as "already done" if the input matches.

4. Decide on Repeated Payloads Up Front: If two genuine runs can carry identical bodies, set idempotencyFields or pass a unique clientIdempotencyId per call — otherwise the second run is deduplicated away. (forceNewExecution exists only on test runs and the streaming execute call.)

5. Make the Key Part of the Template: If a developer has to remember to add an idempotency key, they will eventually forget. Put it in the Gateway configuration you reuse for that API.

Conclusion: Identity is the Backbone of Truth

In a distributed system, you cannot trust the network, you cannot trust the clock, and you cannot trust the response. The only thing you can truly trust is Identity.

SchemaBridge gives you identity at the edges it controls — content-derived execution IDs, explicit dedup keys, and step-level guards — and is honest about the edge it doesn't: the third-party side effect. Derive a stable key from your business data, send it with every call, and the "Double-Charge Nightmare" becomes a design decision you made once rather than an incident you discover later.

Part of our Building the Bridge series. See also: Mastering the Merge on synchronizing state across parallel branches.