Mastering Fan-outs: Building Durable, Idempotent Workflows

SchemaBridge Team · 2025-12-15 · Scalability, Idempotency, Orchestration

Handling large fan-outs without losing track of work. A deep dive into the Spawner vertex, per-item idempotency keys, and at-least-once delivery.

The Large Fan-out Challenge: Where Loops Go to Die

Every developer has written a loop. Whether it's a for loop in Java, a .map() in JavaScript, or a list comprehension in Python, the logic is the same: take a list of items and do something for each one. This is the simplest form of data processing. At the scale of a handful of items, it's trivial. At the scale of a few hundred, it's manageable. But as you cross the threshold into the many thousands, the humble loop becomes a serious liability for your application's reliability and scalability.

In the world of distributed systems, the local loop is a point of failure. When you move from a handful of items to many thousands, the complexity doesn't just increase linearly; it hits a complexity wall. This wall is built from the cold, hard realities of memory management, network latency, and the inevitable failure of the machines running your code. You stop thinking about "logic" and you start fighting "physics."

The Limits of the Local Loop: Why `Promise.all` is Pre-Scale

In a naive implementation, you might receive a massive JSON payload—say, a large daily CSV of orders or a batch export from a CRM—and wrap a loop around an API call to a downstream service. If you're a modern JavaScript developer, you might use Promise.all() to fire them all off in parallel. This is the first mistake of the pre-scale developer.

At scale, this is a disaster for three critical reasons:

1. Memory Exhaustion: The Silent Crasher

Loading a very large number of complex objects into memory can easily crash your worker. Even a modest per-object size adds up quickly, and the raw data can balloon further in memory-heavy runtimes. This is an instant "Out of Memory" (OOM) error that kills the process before the first item is even processed.

2. Execution Timeout: The Clock is Ticking

Most platforms have strict limits. If your loop performs many thousands of API calls, and each call takes just 100ms, your script can run for many minutes. Even if you parallelize it, you are still bound by the resource limits and overhead of that single container. You will be terminated by the platform before the last item is processed, leaving your system in an indeterminate state.

3. No Record of Progress

When the process dies, the only record of what was done lives in the destination system. Nobody can say which items were sent, which succeeded and which were never attempted without forensic reconciliation.

Enter the Spawner: Fan-out as Durable Child Executions

At SchemaBridge, fan-out is a dedicated Spawner Vertex. A Spawner isn't just a loop; it turns each item into a first-class unit of work that the engine tracks, reports on, and lets you retry individually.

How a Spawner Actually Works: The Parallel Split

The Spawner decouples the list from the execution of the items. Each item stops being an iteration inside one process and becomes its own execution:

1. One Child Execution per Item: For every item, the Spawner starts a child execution of the target workflow. Each child is persisted in PostgreSQL with its own ID, its own status and its own logs, and carries its lineage back to the parent run.

2. Deterministic per-item Keys: Every child request carries an idempotency key derived from the parent execution, the Spawner vertex and the item's position. If the Spawner step itself is redelivered—say, its worker pod is restarted mid-fan-out and another worker reclaims the step—it re-submits the list from the start, and children that were already started are recognised by their key instead of being created twice.

3. Lifecycle Independence: If the 501st item fails, it doesn't stop the 502nd from being dispatched or from succeeding. You gain the granularity of many individual executions instead of one massive, brittle batch, and a failed child can be retried on its own.

Be clear-eyed about the guarantee: the dedup on that key is a lookup before the child is created, not a database uniqueness constraint, so it narrows the duplicate window rather than closing it. The engine is at-least-once, and anything a child does to the outside world should be idempotent at the destination.

The Economics of Parallelism

Running a heavy, single-threaded worker for 15 minutes is expensive. Not only are you paying for the high-memory instance, but you are also paying for the "Idle Time" while your code waits for API responses. You are wasting billable CPU cycles on network I/O wait times.

With per-item child executions, the shape of the work changes:

Parallel Execution: Children are picked up by whichever workers have capacity, instead of all running inside one long-lived process.
Reduced Blast Radius: A failure in one child doesn't affect the others. In a traditional loop, a single memory leak or unhandled exception in one item can kill the entire run.

We would expect this to use compute more efficiently than a long-running batch script that mostly waits on I/O, but the bigger win is reliability and observability, not the bill.

The model has bounds, and you should know them. A fan-out is checked against your plan's execution quota before any child is started, because every item is an execution. The Spawner holds the item list in memory and submits children one at a time from a single step, so it is not an infinitely wide pipe. And when the platform's payload store is under pressure, the Spawner defers the whole fan-out—parking the step and re-driving it later from a durable timer—rather than pushing more data into a struggling store.

Idempotency in Distributed Fan-outs: At-Least-Once, Guarded

The biggest hurdle in fan-out patterns is Idempotency. If the Spawner step is interrupted and re-run, how do we avoid starting the same child twice?

SchemaBridge layers several guards:

Per-item Keys: The deterministic key described above lets a re-run Spawner find children that already exist instead of creating new ones. The lookup returns an existing child whatever its status, so a child that already failed is re-attached, not re-run.
Step-level Guards: Inside the engine, a step that has already completed is not re-executed, and claiming a step for execution is an atomic state transition, so a duplicate message for the same step is discarded rather than run twice.

What this does not give you is exactly-once execution. A worker can crash after a side effect has happened but before its completion is recorded, and the step will then run again. That is why we describe the platform as at-least-once with strong guards, and why the destination of any non-idempotent call needs its own protection.

Illustrative Scenario: The Large-Migration Failure

Imagine a team performing a critical migration of a very large batch of ledger entries. They choose to use a traditional Python script. Halfway through a long-running job, a VPN drop occurs. The script crashes.

The Recovery (The Expensive Way)

Recovery is painful: engineers spend days scanning the destination database and manually reconciling records to work out exactly what did and didn't get written.

The Recovery (The SchemaBridge Way)

Now picture the same migration modelled as a Spawner fanning out one child execution per ledger entry, and the same connectivity drop occurring.

1. Failures Stay Per-item: Children that were already running keep going; the children that could not reach the destination fail individually, each with its own error, while the rest of the fan-out is unaffected. If the Spawner itself cannot submit a child, the Spawner step fails visibly rather than pretending to wait.

2. Visible Progress: The run view shows how many children succeeded and how many failed, so the team would not need to reverse-engineer progress from the destination database.

3. Targeted Recovery: Once connectivity is back, the team retries the failed children individually instead of re-running the entire migration. Retrying the failed Spawner step would not re-run them: the per-item keys re-attach every child that already exists, failed ones included, so a Spawner retry only creates the children that were never started. Recovery would become an operator action measured in clicks, not a forensic project—though the ledger writes themselves still need to be idempotent, because a retried child may repeat a write that had already landed.

Detailed Comparison: Scaling Models in the Wild

| :--- | :--- | :--- | :--- |

Expert Checklist for High-Volume Orchestration

If you are designing a high-volume fan-out, follow these rules of thumb:

1. Bound Your Concurrency at the Destination: Protect the downstream system with a concurrency limit. In SchemaBridge that is a per-destination concurrency limit (set on the Rate Limits page), which applies to every step that calls that destination, rather than a knob on the Spawner itself. Start conservatively and widen it as you watch the downstream health.

2. Assume Item Failure is Normal: Ensure each item can be retried independently, and decide up front whether one failed child should fail the whole fan-out or be tolerated.

3. Monitor the "Long Tail" Latency: Look for the small fraction of items that take far longer than the rest. These are usually your most complex edge cases or database lock targets.

4. Leverage Deterministic IDs: Use the engine's per-item keys to protect against restarts, and give your destination a stable business key too. Never rely on a timestamp for uniqueness in a high-concurrency fan-out.

{
  "type": "MERGE",
  "config": {
    "tolerateChildFailures": ["FAILED"],
    "failureTolerance": { "maxFailureRatio": 0.05 },
    "failFast": true
  }
}

Partial-failure tolerance is opt-in and authored on the fan-out's callback merge: by default any failed child fails the join, which is the safe choice when every item matters. Above, failed children are tolerated up to 5% of the fan-out, and failFast settles the join as soon as that budget is definitively exceeded instead of waiting for the stragglers.

Conclusion: Scaling is an Infrastructure Problem

Mastering fan-outs isn't about writing better loops; it's about architecting for distribution. By moving iteration, per-item state and duplicate suppression into the engine, a partial failure becomes a list of specific failed items rather than an unknown state—and that is the difference between a retry and a reconciliation project.

Scaling still has limits—quotas, backpressure and the at-least-once nature of distributed delivery—but those limits are visible and bounded rather than discovered in production at 3am.

See also: The Idempotency Engine: Ensuring Transactional Integrity, for how keys and destination-side idempotency fit together, and Mastering the Merge, for collecting the results of a fan-out.