Executive Summary
Kafka became the default backbone for event streaming, which makes Kafka-API compatibility and who operates the cluster bigger decisions than the broker brand on the box.
The Kafka protocol won the transport layer, so most of this market now competes on operating model and economics rather than the wire format. Confluent (the company behind Kafka, which acquired the diskless WarpStream in 2024), Amazon’s MSK and Kinesis, Azure Event Hubs, Google Pub/Sub, and Kafka-compatible challengers like Redpanda and StreamNative all move and store events — while stream-processing engines such as Apache Flink, now also a Confluent product after the Immerok acquisition, and Databricks handle the separate job of computing over those streams once they flow.
This guide provides a vendor-neutral evaluation framework for 8 leading platforms, separating streaming transport from stream processing, weighing self-managed against fully managed and serverless operations, and pricing the new diskless, object-storage-backed architectures — so you can build event-driven infrastructure you can actually run and afford at production volume, not just demo in a proof of concept.
Why Streaming Data & Event Processing Matters for Enterprise Strategy
The first thing to separate in streaming selection is the transport backbone from the processing layer — brokers move and store events, while engines like Flink compute over them, and they are different decisions. The next is operating model: self-managed Kafka is powerful but operationally heavy, so weigh managed services and Kafka-compatible alternatives against the engineering cost of running clusters, partitions, and retention at scale.
Kafka-protocol compatibility, fully managed and serverless streaming, and tighter integration between streaming and processing are reshaping how teams buy real-time infrastructure. Weigh each platform on operational simplicity, ecosystem compatibility, and cost at your throughput and retention, because streaming is a long-lived backbone whose real expense and toil show up at production scale, not in the proof of concept.
Architecture & Sourcing Decision
Streaming is rarely a literal build-vs-buy question — almost no one writes a broker from scratch. The real decisions are which layer you are buying (transport versus processing), who operates it (self-managed open-source Kafka, a managed cloud service, or the newer bring-your-own-cloud and diskless models), and how tightly you want to bind to one hyperscaler’s ecosystem. Frame the choice around throughput, latency tolerance, and team capacity, not a feature checklist.
| Your Situation | Recommended Path | Rationale |
|---|---|---|
| Mainstream event backbone wanting Kafka without the on-call burden | Fully managed Kafka service | Confluent Cloud, MSK, or a managed Kafka removes ZooKeeper-era toil (Kafka 4.0 is KRaft-only) and partition/retention operations, letting a small team run a production backbone the Kafka ecosystem already supports. |
| High-volume, latency-tolerant logs, telemetry, or data-lake feeds | Diskless / object-storage-backed Kafka | WarpStream, Confluent Freight, Redpanda Cloud Topics, and StreamNative Ursa write through S3-class storage and skip costly inter-AZ replication — the right fit when you can trade single-digit-ms latency for materially lower cloud spend. |
| All-in on one hyperscaler with a lean platform team | Native cloud service (Kinesis, Event Hubs, Pub/Sub) | The cloud-native options bind tightly to IAM, networking, and downstream analytics, minimizing integration work — at the price of portability, since Kinesis and Pub/Sub do not speak the Kafka protocol. |
| Complex stateful computation: joins, windowing, exactly-once | Add a dedicated processing engine (Flink) | Brokers move events; they do not compute over them. Pair the transport layer with Flink (Confluent Cloud, Ververica, or self-managed) or Spark Structured Streaming when the logic outgrows simple consumers. |
| Streaming feeding a lakehouse as the analytical destination | Lakehouse-native streaming (Databricks, Ursa, Iceberg topics) | Where the stream’s job is to land queryable data, engines that write Kafka topics directly to Iceberg or Delta tables collapse the pipeline and avoid a separate copy-into-the-lake step. |
Key Capabilities & Evaluation Criteria
Weight these domains against your own latency targets, throughput, and team capacity. For most enterprises, operating model and total cost at scale now outrank raw feature breadth — the Kafka API has commoditized much of the transport layer, so the differentiators have moved to who runs the cluster, what it costs at your volume, and whether the platform also solves stream processing.
| Capability Domain | Weight | What to Evaluate |
|---|---|---|
| Throughput, Latency & Durability | 25% | Sustained throughput and tail latency at your partition count, ordering and exactly-once / at-least-once delivery guarantees, replication and multi-AZ durability, retention limits, and behavior under broker failure and rebalancing |
| Operating Model & Architecture | 20% | Self-managed vs. fully managed vs. serverless vs. bring-your-own-cloud, KRaft (post-ZooKeeper) operations, diskless / object-storage-backed options, autoscaling, partition rebalancing, and upgrade and patching burden on your team |
| Kafka-API Compatibility & Portability | 20% | Fidelity of the Kafka protocol (consumer groups, transactions, compaction), reuse of existing clients, connectors and tooling, lock-in to a single cloud, and the realistic effort to migrate in or out |
| Stream Processing & Ecosystem | 15% | Native or integrated processing (Flink, Kafka Streams, Spark Structured Streaming, ksqlDB), connector catalog (CDC, sinks, sources), schema registry and governance, and direct landing into Iceberg / Delta lakehouse tables |
| Security, Governance & Compliance | 10% | Encryption in transit and at rest, mTLS / SASL / OAuth and RBAC, audit logging, private networking, data residency and BYOC data-boundary control, and SOC 2 / ISO 27001 / GDPR coverage |
| Cost Model & Economics at Scale | 10% | Pricing unit (capacity / throughput / consumption), inter-AZ and egress charges, storage and retention cost, processing billed separately, and how the bill behaves as volume, partitions, and connectors grow |
Vendor Landscape
The market sorts into four overlapping camps. The Kafka standard-bearer and its ecosystem (Confluent, now including diskless WarpStream and managed Flink) set the protocol and feature pace. The hyperscalers offer both managed Kafka and their own native services — AWS (MSK + Kinesis), Azure (Event Hubs), and Google (Managed Service for Apache Kafka + Pub/Sub) — trading portability for ecosystem gravity. Kafka-compatible challengers (Redpanda, StreamNative/Pulsar) re-architect the engine for lower operational weight and object-storage economics. And the processing layer (Apache Flink, Databricks) computes over the streams the brokers move. Most shortlists end up comparing across these camps, not within one.
Strengths: Built and stewards Apache Kafka, with the deepest ecosystem — Connect, Schema Registry, ksqlDB, and now managed Apache Flink (via the Immerok acquisition) and the diskless WarpStream BYOC engine (acquired 2024). Confluent Cloud spans fully managed, self-managed Platform, and BYOC, and its Freight clusters add object-storage-backed economics for latency-tolerant workloads. Considerations: Consumption pricing across clusters, connectors, processing, and governance can climb quickly and is hard to forecast at fan-out; the platform’s breadth carries a learning curve; and the most attractive economics increasingly live in BYOC/Freight rather than the classic dedicated clusters, so edition choice matters.
Strengths: Two complementary AWS services: MSK runs managed open-source Kafka (with Express brokers for faster scaling and recovery), while Kinesis Data Streams offers a serverless, Kafka-free option with on-demand capacity. Both bind tightly to IAM, VPC, Lambda, Firehose, and the broader AWS analytics stack, minimizing integration effort for AWS shops. Considerations: Strong AWS lock-in; Kinesis does not speak the Kafka protocol, so portability is limited and migration means rewrites; MSK still leaves you owning Kafka sizing and topic operations; cross-region and multi-cloud topologies are comparatively awkward.
Strengths: A fully managed event-ingestion service that exposes a Kafka protocol endpoint, so many Kafka clients connect without code changes. Tiered Standard/Premium/Dedicated capacity, native Azure integration (Functions, Stream Analytics, Fabric), Schema Registry, and Capture to storage make it the path of least resistance for Microsoft-aligned estates. Considerations: Kafka compatibility is at the protocol surface, not a full Kafka cluster — advanced features like Kafka Streams and transactions are newer or in preview, and some ecosystem tooling assumes real Kafka internals; deepest value is inside the Azure ecosystem, with the usual lock-in trade-off.
Strengths: Pub/Sub is a globally scalable, fully serverless messaging service with no partitions or capacity to manage and seamless ties to BigQuery, Dataflow, and Cloud Run; Google Cloud Managed Service for Apache Kafka covers the portability case for teams that need the actual Kafka protocol. Together they let GCP shops choose operational simplicity or Kafka compatibility. Considerations: Pub/Sub is proprietary and behaves more like a queue than a replayable log, so it is not a drop-in for Kafka semantics and ties you to GCP; the Managed Kafka service is younger than AWS and Azure equivalents; running both adds a decision and a learning curve.
Strengths: A Kafka-API-compatible broker rewritten in C++ with a thread-per-core design and no JVM or ZooKeeper, aimed at lower tail latency and simpler operations. Ships as self-managed, BYOC, and Serverless, with Iceberg Topics (GA on AWS and GCP) and Cloud Topics writing directly to object storage for diskless economics, plus the bundled Redpanda Connect integration layer. Considerations: Smaller install base and ecosystem than Confluent; Kafka compatibility is very high but not byte-for-byte identical at every edge; the newest object-storage and lakehouse features are still maturing across clouds; at extreme scale you still need streaming expertise on staff.
Strengths: Pulsar separates serving from storage (BookKeeper) and unifies streaming with queuing, tiered storage, and strong multi-tenancy and geo-replication. StreamNative, founded by Pulsar’s creators, delivers it managed and adds the Ursa engine — a Kafka-API-compatible, lakehouse-native streaming layer that writes topics straight to Iceberg or Delta and earned a VLDB 2025 best-industry-paper award. Considerations: Pulsar’s multi-component architecture (brokers plus BookKeeper plus metadata) is more involved to self-operate than a single-binary broker; the open-source community and talent pool are smaller than Kafka’s; much of the strongest tooling and the Ursa engine are concentrated in StreamNative’s commercial offering.
Strengths: The reference engine for stateful stream processing — event-time semantics, windowing, joins, and exactly-once on large state — and the de facto choice when logic outgrows simple consumers. Available fully managed and serverless on Confluent Cloud (from the Immerok acquisition), as Ververica’s enterprise platform from Flink’s original creators, or self-managed open source, with SQL, Python, and Java/Scala APIs. Considerations: Flink is a processing layer, not transport — it needs a broker (Kafka or compatible) in front of it; self-managed state, checkpointing, and tuning carry a steep learning curve; managed Flink (CFU-based on Confluent, subscription on Ververica) adds cost on top of the streaming bill.
Strengths: Brings streaming and batch together on the lakehouse: Spark Structured Streaming plus Lakeflow Declarative Pipelines (formerly Delta Live Tables) handle incremental ingestion and transformation with built-in orchestration and data-quality checks, and a real-time mode for sub-second latency. Streams land directly in Delta and govern through Unity Catalog, collapsing the stream-to-analytics pipeline. Considerations: Optimized for streaming that feeds analytics and ML on the lakehouse, not as a low-latency message bus — it consumes from Kafka/Kinesis/Event Hubs rather than replacing the broker; micro-batch heritage means the very lowest latencies need the newer real-time mode; spend is tied to Databricks compute consumption.
Pricing Models & Cost Structure
Streaming pricing splits along the operating model, and the unit of measure — provisioned capacity, throughput, or pure consumption — matters more than the headline rate because it decides how the bill behaves as you fan out. Two cost drivers ambush most budgets: inter-AZ replication traffic (exactly what diskless architectures target) and stream processing billed separately from transport. Model cost against your real partition count, consumer fan-out, retention, and processing load, not a single-stream demo.
| Vendor | Pricing Model | Relative Tier | Key Cost Drivers |
|---|---|---|---|
| Confluent | Consumption: elastic/dedicated capacity units + ingress/egress + connectors + Flink + Tableflow | Moderate–Premium | Cluster type (Basic/Standard/Enterprise/Dedicated/Freight), throughput and egress, connector and Flink usage, governance; Freight/BYOC lower inter-AZ cost |
| Amazon MSK & Kinesis | MSK: per broker-hour + storage; Kinesis: per-shard or on-demand throughput | Moderate | Broker size/count and storage (MSK) or shard/throughput and PUT payload units (Kinesis), data retention, cross-AZ transfer, plus downstream AWS service charges |
| Azure Event Hubs | Capacity tiers: throughput units (Standard) / processing units (Premium) / dedicated capacity units | Moderate | Tier and unit count, ingress events and throughput, retention and Capture, namespace dedication; Premium/Dedicated for isolation and advanced Kafka features |
| Google Pub/Sub & Managed Kafka | Pub/Sub: consumption by data volume; Managed Kafka: provisioned vCPU/RAM + storage | Moderate | Throughput and message volume, storage and retention, egress; Pub/Sub bills by data processed, Managed Kafka by provisioned cluster capacity |
| Redpanda | Subscription by capacity/throughput; Serverless consumption; BYOC runs in your cloud account | Moderate | Throughput tier and cluster size, model (Dedicated/BYOC/Serverless), object-storage usage for Cloud Topics, support level; you bear underlying cloud cost in BYOC |
| Apache Pulsar / StreamNative | StreamNative Cloud: capacity/consumption; self-managed open source is free software | Moderate | Throughput and storage (including tiered/object storage), serving vs. BookKeeper resources, Ursa/lakehouse usage; self-managed shifts cost to your own operations effort |
| Apache Flink (Confluent / Ververica) | Confluent: serverless processing units per minute; Ververica: subscription; open source is free | Moderate–Premium | Processing-unit minutes or licensed capacity, parallelism and state size, checkpoint storage; this is a separate bill on top of the streaming transport layer |
| Databricks | Consumption: compute units (DBUs) by workload tier + underlying cloud infrastructure | Moderate–Premium | Cluster size and uptime, workload tier, real-time vs. micro-batch mode, storage and Unity Catalog; cost tracks always-on streaming compute |
Implementation & Migration
Sequence the rollout by criticality and by layer — stand up the transport backbone and prove one real producer-to-consumer flow before layering on stateful processing. Get schema governance and the operating model right early, because a streaming backbone is long-lived and the toil compounds at production scale.
Inventory event sources and consumers, set latency, ordering, and retention requirements per use case, and decide the operating model — self-managed, fully managed, serverless, or diskless/BYOC. Choose Kafka-API compatibility versus a native cloud service, and decide whether you need a separate processing engine. Run a production-shaped POC and read the itemized bill.
Stand up the cluster (KRaft-based, post-ZooKeeper), wire in identity, encryption, and private networking, and establish the Schema Registry and naming, partitioning, and retention conventions. Migrate one real producer and consumer end to end, and codify topic, ACL, and quota standards before breadth arrives.
Add the stream-processing layer (Flink, Kafka Streams, or Spark Structured Streaming) for stateful logic, deploy connectors for CDC and sinks, and land streams into the lakehouse where that is the destination. Validate exactly-once or at-least-once semantics under failure, and rehearse rebalancing and broker loss.
Roll out remaining workloads, tune partitions, consumer fan-out, and retention, and review the bill against the model — targeting inter-AZ and egress hot spots with diskless or tiered-storage options. Establish on-call runbooks, SLOs, and lag monitoring, and retire the batch pipelines the stream now replaces.
Selection Checklist & RFP Questions
Use this checklist during evaluation to ensure each shortlisted platform covers what actually decides a streaming backbone, not a generic feature grid.