Designing for Velocity: The Case for Schema-less Event Ingestion

SchemaBridge Team · 2025-12-08 · Event Ingestion, JSON, DX

Speed vs. strict types in event-driven systems. Why we chose raw JSON over strict Type Definitions.

The Schema Paradox: Friend or Foe?

In traditional enterprise software, strict schemas (SQL, Protobuf, WSDL) are the bedrock of reliability. The contract is simple: I define the shape of the data, you adhere to it, and the compiler guarantees that we won't crash. This approach has served us well for decades in controlled environments where we own both sides of the pipe. It provides compile-time safety, efficient binary serialization, and a clear "source of truth" for developers.

But in a modern, high-velocity integration environment—where SaaS vendors change their payloads weekly, legacy ERPs emit "flexible" CSVs, and internal microservices are born and retired monthly—rigid schemas become a straitjacket. They provide safety at the cost of Inertia. When the world around you is fluid, a rigid contract is not a foundation; it's a point of failure. The Schema Paradox is this: the more you try to protect your system with strict types, the more fragile you make it when the outside world shifts.

The Historical Evolution: From COBOL to JSON

To understand the demand for schema-less systems, we must look at the evolution of data exchange. In the early days of mainframe computing, data was stored in Fixed-Length Records (COBOL COPYBOOKs). If you wanted to add a field, you had to recompile every program that read that record. It was the ultimate rigid schema. This was the era of the "Single Machine Mindset," where the cost of data storage was so high that every byte had to be accounted for in a fixed position. There was no room for hierarchy, no room for optionality, and certainly no room for evolution. Every character in a record was a precious resource, and any change was a seismic event that required weeks of planning and testing.

In the 1980s and 90s, we moved toward Relational Databases and SQL. This was a massive step forward, as it introduced the concept of structured relationships. But it also introduced the Database Migration. Adding a column meant downtime, locking tables, and careful coordination between DBAs and developers. The schema was still a wall that developers had to climb every time they wanted to innovate. Even with the rise of ORMs like Hibernate, the underlying rigidity of the table structure remained the final arbiter of what was possible.

Then came XML and SOAP in the late 90s. This introduced the concept of "Tags," allowing for some flexibility. You could add an XML tag without necessarily breaking the parser. However, the industry quickly added XSD (XML Schema Definition) and WSDLs, which brought back the rigidity. We spent years fighting with namespaces and complex enterprise service buses that would reject a message if a single character was out of place. It was the "Dark Age of XML," where the overhead of the schema consumed more bandwidth than the actual data.

Today, we have JSON and REST. JSON is naturally flexible. It's just a map of keys and values. Yet, our engineering instincts still drive us to wrap that flexibility in strict types (TypeScript interfaces, Java DTOs, Avro schemas). We are trying to impose 1970s mainframe rigidity onto 2020s cloud events. Why? Because we fear the unknown. We fear that a missing field will crash our service. But as we will see, this fear is being addressed by the wrong tools.

The Financial Analysis: The "Hidden Tax" of Integration Maintenance

Let's quantify the cost of schema rigidity. In a typical mid-to-large engineering organization, integration maintenance is a "Quiet Crisis." It doesn't appear on a balance sheet as a line item, but it is a massive drag on productivity.

The Math of Maintenance

The pattern is easy to see once you look for it. An organization with a large portfolio of external SaaS integrations will watch vendors change their payloads regularly, and each change that breaks a strict parser sets off the same cycle: detection, DTO/schema updates, testing, and a CI/CD deployment. A handful of hours per break, multiplied across many breaks a year, adds up to a substantial and recurring drain on senior engineering time.

But the real cost is the Opportunity Cost. While your senior engineers are updating DTOs for the latest Stripe update, they aren't building the new automated fraud detection feature that moves the business forward. Over time, this tax compounds into a Cumulative Loss of Velocity that can put a company well behind its more agile competitors.

The Philosophy of Data Fluidity: Schema-on-Read

At SchemaBridge, we advocate for a fundamental change in mindset: Schema-on-Read.

Instead of validating the data at the moment of entry (Schema-on-Write), we ingest the raw, hierarchical truth of the event first. We preserve every byte of the JSON payload in our durable store. We only apply a schema—or more accurately, a Transformation—at the moment the data is needed by a specific business process.

Why Data Fluidity Wins

1. Zero-Touch Ingestion: You can start receiving events from a new provider in seconds. Point the webhook at a SchemaBridge Gateway, and the data begins flowing into the durable store immediately. You can figure out what the data means later.

2. Insurance Against Unknowns: If a provider adds a field today that you don't need, it's still captured in the raw JSON. If you realize six months from now that you do need that field, the historical data is already there. You don't have to go back and ask the provider for old data.

3. Decoupled Evolution: Your ingestion layer and your transformation layer can evolve at different speeds. You can update your business logic 10 times a day without ever touching your ingestion gateways.

The Mathematical Case for Late-Binding

In computer science, Late-Binding is the practice of delaying the resolution of an identity or a type until the moment of execution. This is what makes dynamic languages like Ruby or Python so powerful for certain tasks.

Schema-on-Read is late-binding applied to your data infrastructure. By delaying the mapping, you move from a Rigid Graph (where every change requires a full rebuild) to a Flexible Path (where the path can adapt to the terrain as it moves).

Mathematically, the number of potential mappings between $N$ producers and $M$ consumers is $N \times M$. If every producer and consumer must agree on a strict schema, you have a massive coordination problem. If you use a schema-less bridge with late-binding, you reduce the problem to $N + M$ mappings, where each mapping is local and independent. This is how you achieve True Horizontal Engineering Scale.

JSONata: The Masterclass in Functional Event Processing

To make Schema-on-Read practical, you need a language that is designed for discovery. We chose JSONata. JSONata is not just a query language; it is a functional transformation engine that operates directly on the raw JSON hierarchy.

The Anatomy of a JSONata Expression

Consider a payload from a legacy ERP that returns a list of orders. Each order has a complex, nested structure. You want to extract all part numbers for orders that are higher than $500 and are currently in the 'SHIPPING' state.

The Traditional Code (Javascript):

const parts = payload.orders
  .filter(o => o.total > 500 && o.status === 'SHIPPING')
  .flatMap(o => o.items)
  .map(i => i.partNumber);

This code is fragile. If orders is null, or if items is missing for one order, it crashes.

The JSONata Mastery:

orders[total > 500][status = 'SHIPPING'].items.partNumber

This expression is Null-Safe. If orders is missing, the result is simply an empty array. It never throws an exception. It "discovers" the data rather than "asserting" it.

Advanced Pattern: Deep Descendent Selection

One of the most powerful features of JSONata is the ** operator. It allows you to find any key, no matter where it is in the hierarchy.

$**.tracking_number

If your downstream providers all use different nesting for tracking numbers, this one expression will find them all across every different payload version. This is the definition of Structural Resilience. It turns a brittle hunt for a specific key into a flexible search for the truth.

Advanced Pattern: Data Restructuring on the Fly

JSONata allows you to rebuild the entire JSON object in a single pass.

orders.{ "order_id": ID, "summary": $join(items.name, ', ') }

In traditional code, this requires object mapping, string concatenation, and array iteration. In JSONata, it is a declarative projection of your desired state. This is especially powerful when you need to send a simplified summary of a complex event to a Slack notification or a mobile app.

Operationalizing Table-less Data: Security and Validation

Critics often ask: "If we don't use schemas, how do we prevent garbage from entering our system?"

The answer is that Schema-less != Unvalidated. We simply move the validation to the Vertex Level.

Case Study: The 50-Region Data Ingestion Mesh

Consider a global IoT fleet ingesting telemetry from many regions, each running a slightly different version of its sensor firmware. Every region has its own "dialect" of JSON.

The Challenge

Suppose the team starts with a traditional SQL-based ingestion system and a strict table schema. Every time a firmware update rolls out in a single region, the ingestion pipeline for that region breaks, because the new firmware adds a battery_health_v2 field that the database has no column for. The data team lands in constant "Fire-fighting" mode, running ALTER TABLE commands across many production databases and dropping events during every maintenance window.

The SchemaBridge Solution

Now imagine the same team moves to a schema-less strategy.

1. Universal Capture: Every region points its data at a single SchemaBridge Gateway cluster. The Gateways don't care about the schema; they simply persist the raw events.

2. Mapping on Read: The team creates a "Normalization Vertex" per firmware version. The workflow identifies the firmware version from the header and routes the raw JSON to the correct vertex.

3. No-Code Upgrades: When a new firmware version launches, there's no database to update. The team duplicates the existing vertex, updates the JSONata mapping to include the new fields, and deploys.

The Result

Conclusion: Velocity is a Design Choice

Strict schemas are a choice. They are a choice to prioritize static safety over dynamic velocity. In a closed system, that's a valid choice. In a connected, distributed ecosystem, it's a choice that leads to failure.

Design your systems for the world as it is—unpredictable, evolving, and hierarchical. Design for velocity. Design with SchemaBridge. By embracing the fluidity of data, you unlock the ability to build, scale, and innovate at a speed that your competitors—stuck in their rigid DTOs and database migrations—can only dream of.

Next in this series: moving from ingestion to execution with the "Spawner" vertex — how to master distributed fan-outs that spread high-volume work across many items without crashing your servers or losing a single transaction. Reclaiming the power of the loop in a distributed context.

Explore