Real-time inference matters only when a prediction arrives in time to change an outcome. A fraud score returned after authorization, a pricing signal calculated after the customer leaves, or an equipment alert raised after failure is analytically interesting but operationally late.
The architecture above separates event capture, feature preparation, inference, and action. That separation is deliberate. It allows each stage to scale, fail, and recover independently while preserving the history required to replay a decision and explain which data and model produced it.
Define the decision deadline first
“Real time” is not a design target. A payment decision may have hundreds of milliseconds; an industrial intervention may allow seconds; a recommendation may remain useful for minutes. Write down the end-to-end deadline, the consequence of missing it, and the acceptable degraded behavior. These three facts determine whether inference belongs synchronously in a request path or asynchronously behind an event.
Allocate the budget to each stage: producer, broker, deserialization, feature lookup, model call, policy checks, and result delivery. Leave room for network variance and cold starts. A model benchmark measured in isolation is not the latency users experience.
Treat the event contract as a public interface
Events should carry a stable identifier, event time, schema version, source, tenant or account context, and the minimum business facts needed downstream. Use a schema registry or equivalent compatibility checks. Producers should not be able to change a field’s meaning silently simply because the serialized type still validates.
Distinguish event time from processing time. Events can arrive late or out of order, especially during retries and network interruption. Windowed features must define how late data is handled; otherwise a model can receive a technically valid feature vector that describes the wrong moment.
Build a durable, replayable ingestion layer
Kafka, Amazon Kinesis, and similar streams provide a durable boundary between producers and consumers. Partition keys determine ordering and parallelism, so choose them from the business invariant: card, customer, device, account, or workload. A key that concentrates traffic can limit throughput; a random key can destroy the ordering a feature calculation depends on.
Retention is an engineering control, not cheap archival by default. Keep enough history to recover from a consumer defect and reproduce important decisions. For longer retention, move immutable events to object storage with lifecycle rules. Record offsets or checkpoints alongside deployable consumer versions so replay is controlled rather than improvised during an incident.
Keep online and training features consistent
Many production failures are feature failures: a transformation differs from training, a lookup is stale, a default value changes, or a field is computed over the wrong time window. Define feature logic once where practical, version it, and record the versions used for every inference. An online feature store can reduce lookup latency, but it does not remove the need to test data freshness and parity.
Separate validation from enrichment. Reject or quarantine malformed events before they contaminate state. For missing optional data, use explicit defaults that were represented during training. If a required feature is unavailable, the system should choose a documented fallback—such as a rules-based decision or manual review—rather than passing a misleading zero to the model.
Make inference consumers idempotent
Most event systems are designed around at-least-once processing somewhere in the path. A message can be delivered again after a timeout even if the first attempt succeeded. Use the event identifier and model version to create an idempotency key. Store the decision before acknowledging the message, and make downstream actions reject duplicates.
Control concurrency to protect both the model endpoint and the action system. Backpressure is preferable to uncontrolled retry storms. When the queue grows, the application needs a policy: scale consumers, shed low-priority work, use a cheaper fallback, or expire predictions that can no longer influence the decision.
Design failure paths before the happy path ships
A dead-letter stream should contain enough context to diagnose and replay the event without copying unnecessary sensitive data. Classify failures into transient, data-quality, policy, and model failures. Retry only transient conditions, with bounded attempts and jitter. Invalid schemas and denied policy checks need investigation, not repeated model calls.
Version the model endpoint independently from the consumer. Use shadow traffic to compare a candidate model, then a canary partition or limited audience before broad rollout. Keep the previous version available long enough to roll back, and ensure the event record can identify which model, feature set, threshold, and code version created the result.
Observe one decision across every stage
Carry a correlation identifier from producer to action. Measure event age, consumer lag, feature freshness, inference latency, timeout rate, queue depth, dead-letter volume, and action completion. Technical health must be connected to model health: score distribution, calibration, drift, override rate, and the eventual business outcome.
Begin with a replayable historical stream and run the new pipeline without affecting decisions. Confirm ordering, duplicates, feature parity, and latency under burst conditions. Only then allow the inference result to advise users, and later to drive bounded actions. The durable event history makes this staged rollout possible—and makes failures explainable when production behaves differently from the test set.