Real-time analytics: when streaming data is worth it, and how it works
Real-time analytics answers questions while there is still time to act on them. It also costs more to build and run, so the first job is deciding which questions need it.
Most business reporting runs in batches: data is loaded overnight and dashboards show yesterday. Real-time analytics processes data as events happen, so a decision can be made in seconds or minutes. The value depends entirely on whether anyone, or any system, can act that fast.
Where real-time pays off
The Apache Kafka project lists typical uses for event streaming: processing payments in real time, tracking vehicles and shipments, analyzing sensor data from IoT devices, and responding to customer orders. Common business cases follow the same pattern:
| Use | Why seconds matter |
|---|---|
| Fraud detection | A payment has to be blocked before it completes |
| Inventory and pricing | Overselling stock or missing a demand spike costs money now |
| Operations monitoring | Faults in equipment or services need action before customers notice |
| Customer experience | Recommendations and offers work best during the session |
Monthly financial reporting, board dashboards and most planning work don't need real-time data. Building streaming for them adds cost without changing any decision.
How it works
A real-time setup usually has three parts.
First, events are captured. Applications publish events directly, or change data capture tools read database logs. Debezium, an open source CDC tool, reads the MySQL binlog or a PostgreSQL logical replication stream and turns each change into an event, without changing the application.
Second, events are stored and moved. Kafka describes its job as capturing streams of events from databases, sensors and applications, storing them durably, and processing and routing them. Managed versions are available from the major clouds and from Confluent.
Third, events are processed and served. Stream processors such as Apache Flink or Kafka Streams calculate running totals, detect patterns and join streams. Results go to a fast analytics database, an alert or an application.
Streaming or batch
A practical rule: use streaming where a person or system will act within minutes of an event, and batch everywhere else. Many companies end up with a small streaming layer for a few operational uses and a warehouse or lakehouse for everything else, fed by the same event streams.
What IT teams should do
- List the decisions that would change if data arrived in seconds rather than the next day.
- Start with one use case, such as fraud alerts or stock levels, and measure the result.
- Use change data capture to stream from existing databases instead of rewriting applications.
- Use a managed streaming service unless you have staff to run Kafka clusters.