The Hidden Cost of 'Real-Time' Data: When Batch Beats Streaming

Introduction

"Make it real-time" became a default ask from product and ops teams in 2025. By 2026, the infrastructure bills made everyone notice. Streaming at production scale costs 3-10x batch, and most "real-time" dashboards work just fine on 5-minute batches.

This post breaks down the cost math behind streaming, when it genuinely earns its keep, when batch is the right answer, and the hybrid pattern most mature data platforms converge on. We'll cover specific workloads where real-time is non-negotiable, where batch is dramatically better, and how to right-size freshness per workload instead of treating real-time as a platform-wide aspiration.

If your AWS bill on data infrastructure grew faster than your revenue in the last year, this is the post for you.

The cost math nobody runs

Streaming costs come from three places that batch doesn't pay:

Infrastructure baseline

A production Kafka cluster, Flink job runner, and Materialize instance all carry meaningful fixed cost — even if you're processing zero events. We typically see baselines of $2-5k/month for a moderate streaming stack before any throughput.

Throughput-priced storage and compute

Streaming systems charge per event written, per query consumed, per partition. At 10k events/second sustained, you're looking at $5-20k/month in Confluent Cloud, Materialize, or similar managed offerings — and that's before storage retention and replication.

Engineering operations

Streaming systems require more on-call attention than batch. Schema evolution is harder, monitoring is denser, debugging is more complex. We typically estimate 25-50% more engineering time on a streaming pipeline vs. an equivalent batch pipeline over a 12-month period.

The math example

A typical "real-time customer 360" dashboard at moderate scale costs:

cost-comparison
Streaming version:
  Kafka cluster (managed):     $2,500/mo
  Flink job runner:            $1,800/mo
  Materialize (or similar):    $3,500/mo
  Storage + retention:         $1,200/mo
  ──────────────────────────────────────
  Total:                       $9,000/mo

Batch version (5-min refresh):
  Snowflake compute (X-Small): $400/mo
  Snowflake storage:           $200/mo
  Fivetran ingestion:          $1,200/mo
  ──────────────────────────────────────
  Total:                       $1,800/mo

When streaming actually earns its cost

Streaming is the right answer when seconds matter for business outcomes. The workloads where we genuinely recommend streaming:

Fraud detection and anomaly response

Fraudulent transactions need to be blocked in real time, not analyzed after the fact. The cost of a delayed decision (chargebacks, downstream fraud) far exceeds the streaming infrastructure cost. This is where streaming earns 10x its cost.

Real-time ML inference features

When ML models need features computed at request time (within 100ms of a user action), those features have to be precomputed or streaming-computed. Recommendation engines, ad targeting, dynamic pricing, fraud scoring all live here.

Operational dashboards for active intervention

Call center routing dashboards where supervisors actively redirect call flow. Ad bidding systems where humans monitor and adjust budget allocation. Live ops triage during incidents. The user is acting on the data in real time — staleness directly degrades the action.

Event-driven product features

User-visible product features that depend on freshness: notifications, real-time collaboration, presence indicators, live activity feeds. These cannot work on batched data.

Notice what's not on this list: "an executive wants the dashboard to update faster." That's a UX problem (better caching, smarter polling, optimistic updates) — not a data freshness problem.

When batch is dramatically better

Batch is the right answer when freshness is not the bottleneck. For these workloads, you're paying streaming costs for marginal user benefit:

• Executive dashboards. Revenue, MRR, churn, NPS. These get reviewed daily or weekly. 5-minute batch is plenty fresh; 30-second streaming is wasted spend.
• Financial reporting. Accounting, finance, audit. Data needs to be accurate, complete, and reconcilable — not real-time.
• Historical analysis. Cohort analysis, attribution, customer lifetime value. Inherently retrospective.
• Planning and forecasting. Demand forecasting, capacity planning, budget modeling. Weekly or monthly refresh is fine.
• Most ML training data. Training datasets get refreshed before each training run, not continuously.
• Product analytics for trends. "What's our DAU trend?" "Which features have growing usage?" These read better at hourly or daily granularity than at second-level.

The hybrid pattern most teams land on

Mature data platforms don't choose between streaming and batch — they use both. The pattern that works:

Layer 1: Batch warehouse for everything

Snowflake, BigQuery, or Databricks with dbt transformations on a 5-minute, hourly, or daily refresh schedule depending on workload. This handles 80% of analytics and reporting.

Layer 2: Real-time tables for specific use cases

Materialize, ClickHouse, or RisingWave for the specific dashboards and product features that need sub-minute freshness. These read from the same Kafka stream as the warehouse but maintain real-time materialized views.

Layer 3: Stream processors for actions

Flink, Kafka Streams, or Spark Structured Streaming for the specific transformations that need to drive real-time actions — fraud blocking, ML feature computation, real-time alerting. These produce results that are consumed by application code, not dashboards.

Cost optimization patterns we use

For clients with already-running streaming infrastructure that's eating budget, we typically find 30-50% cost reductions through these patterns:

• Tiered freshness: Real-time for the small subset of dashboards that need it; batch for everything else. Often 80%+ of streaming costs are wasted on workloads that don't need it.
• Smarter retention: Most streaming systems retain data far longer than needed. Tier into cheaper storage (S3) after 7-30 days.
• Compute scaling: Many streaming workloads are bursty. Auto-scaling Flink jobs or Materialize replicas based on lag can dramatically reduce idle cost.
• Smarter consumption: Streaming queries that read the same data repeatedly should be cached or materialized once. We see significant cost in re-computing the same aggregation.
• Aggregation pushdown: Aggregate as early as possible in the pipeline. Querying raw event streams is expensive; querying pre-aggregated tables is cheap.

How to decide for any specific workload

When evaluating freshness needs for a new dashboard or feature:

• What's the user doing with the data? Acting on it in real time → consider streaming. Reading it for context → batch is fine.
• How fresh does it actually need to be? If "5 minutes vs 30 seconds" is indistinguishable from the user's perspective, batch wins.
• What's the cost of stale data? Quantify it. If the cost of acting on 5-minute-old data is essentially zero, you don't need streaming.
• What's the cost of the streaming infrastructure? Calculate it. The number is often surprising.
• Can the use case be redesigned? Sometimes UX changes (better caching, optimistic updates, partial data) deliver the perceived freshness without the streaming cost.

Conclusion

Default to batch. Add streaming where the business case clearly justifies the cost. Measure baseline freshness needs per workload, not as a platform-wide aspiration. Build the hybrid stack that uses each tool where it fits.

Your future self and your CFO will thank you. So will the on-call rotation that no longer gets paged at 3am because the streaming pipeline lagged on a workload that didn't need to be streamed in the first place.

If your team is fighting infrastructure costs and isn't sure which workloads to migrate from streaming to batch, we run focused cost audits that typically identify 30-50% reduction opportunities in 4-6 weeks. Worth knowing where the money's actually going before you cut.