Executive Summary
Decide three things before comparing products: which protocol your applications speak, who will run the cluster, and whether you are buying a place to keep events or an engine to compute over them.
A data streaming platform captures events as they happen, keeps them in order, and lets many applications read and act on them, either at once or later. The Apache Kafka project defines event streaming as the practice of capturing data in real-time from event sources like databases, sensors, mobile devices, cloud services, and software applications in the form of streams of events; storing these event streams durably for later retrieval; manipulating, processing, and reacting to the event streams in real-time as well as retrospectively; and routing the event streams to different destination technologies as needed. Choose a platform on three things: the protocol your producers and consumers use, who operates the cluster, and whether you also need an engine that computes over the streams. The Apache Kafka project states that you can choose between self-managing your Kafka environments and using fully managed services offered by a variety of vendors.
The Apache Kafka project describes Kafka as an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Most products in this guide either run Apache Kafka or accept Kafka clients, so the protocol rarely separates them. What separates them is where the data is stored, who is on call, how the bill is counted, and how the streams reach your analytics.
Event stream processing is a separate decision. The Apache Flink project describes Flink as a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. A streaming platform stores and delivers events; a stream processor joins, aggregates and reacts to them. Some vendors sell both, and the same processing engine can read from several brokers, so price and test each half on its own.
Why the Operating Model Decides the Cost
Streaming costs are set less by license price than by how data moves between machines. A Kafka cluster spread across availability zones copies each partition to brokers in other zones so it survives the loss of one, and cloud providers can charge for that traffic. Apache Kafka notes that two out of the 3 major clouds charge for cross-availability-zone traffic. Google Cloud states that a Managed Service for Apache Kafka cluster replicates data across multiple zones, and charges for inter-zone data transfer within the cluster at $0.01 per 1 gibibyte. The charge grows with every gigabyte written and every extra copy, so estimate it from your own throughput and replication factor.
The open-source project is changing its storage model in response. Apache Kafka lists KIP-1150: Diskless Topics with a current state of Accepted. Apache Kafka states that diskless topics allow Apache Kafka operators on hyperscalers to eliminate inter-zone data transfer costs from replication. Accepted means the design is agreed, not that it ships in a release you can run today. Several commercial products already write streams directly to object storage, and the vendor section names them.
Operations have also become simpler. The Apache Kafka project describes Kafka 4.0 as the first major release to operate entirely without Apache ZooKeeper, running in KRaft mode by default. Clusters still running on ZooKeeper face a migration before they can move to current releases, which is a reason to price a managed service against an upgrade project rather than against an unchanged cluster.
Kafka is no longer only a log. The Apache Kafka project announced early access to Queues for Kafka (KIP-932), enabling Kafka to support traditional queue semantics directly. Queue semantics let one platform handle work distribution as well as event fan-out, though early access is not a production commitment, so check the status in the release you would deploy.
Streams and analytics are also converging. Products now write topics straight into open table formats such as Apache Iceberg and Delta Lake, so analysts can query recent events without a separate loading job. Ask each vendor whether the tables are written by the broker, by a connector you operate, or by a separate paid service, because that decides who fixes them when the schema changes.
Streaming Platform or Stream Processor: What You Are Buying
The Apache Kafka project states that Kafka combines three key capabilities: to publish (write) and subscribe to (read) streams of events, to store streams of events durably and reliably for as long as you want, and to process streams of events as they occur or retrospectively. Commercial products split those jobs differently. Some sell the event store and its connectors, some sell a processing engine, and some sell both under one bill.
Almost nobody writes a broker from scratch. The real choice is who runs it. Self-managed open-source Kafka costs no license fee and needs a team that can patch, rebalance and recover it. A managed service runs in the vendor’s cloud account and charges for capacity or throughput. A bring-your-own-cloud service runs in your account under the vendor’s management, so data stays in your network and you pay the cloud provider for the machines and storage.
Messaging services sit next to this category. A queue hands each message to one worker and deletes it when the work is done. An event stream keeps every event for a retention period so new consumers can replay history. Many products now offer both models, so describe the workload in those terms when you ask for a quote.
| Need | What it answers | What to buy |
|---|---|---|
| Event store | Where events are written, kept and replayed from, and by whom | A streaming platform, managed or self-managed |
| Stream processing | How events are joined, aggregated and turned into results as they arrive | A processing engine or managed service, alongside the event store |
| Connectors | How databases, SaaS applications and storage feed and receive streams | The platform’s connector service, or connectors you run yourself |
| Streams as tables | How analysts query recent events with SQL | Topic-to-table features, or a lakehouse that reads the stream |
| Work queues | How one task reaches exactly one worker | A messaging service, or a streaming platform with queue semantics |
How do you evaluate Data Streaming Platforms & Event Stream Processing?
Protocol compatibility comes first. A service that speaks the Kafka protocol lets existing clients connect by changing a connection string, but compatibility varies by feature. Ask for the Kafka client versions supported, and whether transactions, Kafka Streams, compression and the admin API work as they do on Apache Kafka. Test with your own clients and with the monitoring and admin tools that connect to the cluster.
Storage design sets the cost and the latency. A cluster that keeps data on broker disks and replicates it between zones has low latency and pays for cross-zone traffic. A design that writes straight to object storage avoids that traffic and accepts higher latency. Tiered storage sits between the two: recent data on disk, older data in object storage. Ask each vendor where a write is acknowledged, what latency it promises at the 99th percentile, and what it charges per gigabyte written, stored and read.
Retention and replay decide what a new consumer can do. Ask how long data can be kept, how it is priced beyond the default period, and whether reading old data is charged separately.
Processing is the second half of the platform. Ask whether stream processing runs on the same service or is billed separately, which languages it accepts (SQL, Java, Python), how it manages state and recovers from failure, and whether it can read from a broker run by another vendor.
Governance travels with the data. Check for a schema registry, access control per topic, audit logs, private networking and encryption with your own keys, and which pricing tier includes each.
| Capability | What it does | Buyer translation |
|---|---|---|
| Kafka protocol compatibility | Lets existing Kafka clients and tools connect | Which client versions and features are supported, and which are in preview |
| Object-storage design | Writes streams to object storage instead of broker disks | Lower cross-zone cost in exchange for higher latency |
| Tiered storage | Moves older segments to cheaper storage | Longer retention without larger brokers; check read charges |
| Stream processing | Joins, aggregates and filters events as they arrive | Same bill or separate service, and which languages |
| Topic-to-table | Writes topics as Iceberg or Delta tables | Who maintains the tables when schemas change |
| Deployment model | Vendor cloud, your cloud account, or your own servers | Where data lives and who is on call |
Vendors in Data Streaming Platforms & Event Stream Processing
Representative vendors include Amazon MSK & Kinesis, Apache Flink / Confluent / Ververica, Apache Pulsar / StreamNative, Azure Event Hubs, Confluent, Databricks, Google Pub/Sub & Managed Kafka and Redpanda. The list is alphabetical, is not a ranking, and is not for sale. How we choose vendors
The approaches below follow two questions: what software does the product run, and who operates it? Most of the field speaks the Kafka protocol, so those two answers separate the vendors more than features do.
The Kafka platform from its co-creators comes first. Confluent lists Jun Rao as a co-founder of Confluent and original co-creator of Apache Kafka. IBM announced on March 17, 2026 that it had completed its acquisition of Confluent. Confluent announced that it has acquired WarpStream, a Kafka-compatible streaming solution with a unique architecture. WarpStream describes itself as a diskless, Apache Kafka-compatible data streaming platform built directly on top of object storage, with zero inter-AZ costs, that runs in your VPC.
Each major cloud provider sells its own streaming services. AWS describes Amazon Managed Streaming for Apache Kafka (Amazon MSK) as a streaming data service that manages Apache Kafka infrastructure and operations. AWS describes MSK Serverless as a cluster type that runs Apache Kafka without having to manage and scale cluster capacity. AWS describes Amazon Kinesis Data Streams as a serverless, real-time data streaming service to capture, process, and analyze data streams at any scale. Microsoft describes Azure Event Hubs as a fully managed, real-time data streaming platform, a native Azure service with built-in Apache Kafka compatibility that runs existing Kafka workloads without code changes or cluster management overhead. Microsoft states that the Kafka endpoint in Event Hubs is supported only in the standard, premium, and dedicated tiers. Google Cloud describes Pub/Sub as an asynchronous and scalable messaging service that decouples services producing messages from services processing those messages. Google Cloud states that Pub/Sub uses per-message parallelism, rather than partition-based messaging. Google Cloud describes Managed Service for Apache Kafka as a service that helps you run secure, scalable open source Apache Kafka clusters. Google Cloud states that Managed Service for Apache Kafka runs in KRaft mode and that ZooKeeper mode is not supported.
Two approaches rebuild the broker rather than run Apache Kafka. Redpanda describes Redpanda Streaming as a Kafka-compatible, C++ engine with zero dependencies. Redpanda states that it is packaged as a single binary that doesn't rely on any external systems. Redpanda offers three fully-managed deployment options for Redpanda Cloud: Serverless, Dedicated, and Bring Your Own Cloud (BYOC). The Apache Pulsar project describes Pulsar as a multi-tenant, high-performance solution for server-to-server messaging, originally developed by Yahoo and now under the stewardship of the Apache Software Foundation. Apache Pulsar has native support for multiple clusters in a Pulsar instance, with geo-replication of messages across clusters. StreamNative states that its Pulsar clusters support both Pulsar and Kafka-compatible protocols. StreamNative states that its BYOC (Bring Your Own Cloud) clusters are hosted on your cloud, and fully managed by StreamNative. StreamNative states that Kafka and Pulsar run on Lakestream, the open stream storage implemented by Ursa, so streams sit next to Iceberg and Delta tables on object storage without a separate stream-to-table pipeline.
Processing has its own vendors. Apache Flink states that unbounded streams have a start but no defined end. Ververica states that it created Apache Flink. Ververica describes itself as an enterprise platform for Apache Flink, deployed as BYOC in your own AWS or Azure account or self-managed on your own infrastructure. AWS states that Amazon Managed Service for Apache Flink lets you transform and analyze streaming data in real time using Apache Flink. The lakehouse is the other route. Databricks states that Apache Spark Structured Streaming lets you build streaming applications and pipelines with the same Spark APIs you use for batch processing. Databricks states that Apache Kafka can be used as either a source or a sink for Structured Streaming workloads on Databricks. Databricks states that real-time mode is designed for sub-second operational workloads, with end-to-end latency as low as five milliseconds. Databricks states that micro-batch processing is the default mode.
| Vendor | Approach | Where it fits |
|---|---|---|
| Amazon MSK & Kinesis | Cloud provider streaming services | Managed Apache Kafka, serverless Kafka and Kinesis Data Streams on AWS |
| Azure Event Hubs | Cloud provider streaming services | Native Azure event ingestion with a Kafka endpoint from the Standard tier up |
| Google Pub/Sub & Managed Kafka | Cloud provider streaming services | Per-message messaging, or open-source Kafka clusters run by Google Cloud |
| Confluent | Kafka platform from its co-creators | Managed and self-managed Kafka, with WarpStream for diskless streaming in your own cloud; owned by IBM |
| Redpanda | Kafka-compatible engines | A C++ engine for Kafka clients, self-managed or in Serverless, Dedicated and BYOC clouds |
| Apache Pulsar / StreamNative | Multi-tenant messaging and streaming | Multi-tenant Pulsar with Kafka-compatible protocols, managed in its cloud or yours |
| Apache Flink / Confluent / Ververica | Stream processing engines | Open-source stateful stream processing, with commercial platforms from Ververica and Confluent |
| Databricks | Streaming inside the lakehouse | Structured Streaming and Lakeflow pipelines that read from and write to Kafka |
Eight vendors are named across six approaches, and three of them share the cloud provider approach. The approaches were set out before any vendor was selected, and a row here cannot be bought. The field is wider, including independent managed Kafka services and other diskless engines, and any streaming vendor can make its own case in the Spotlight below.
How much should you budget for Data Streaming Platforms & Event Stream Processing?
Streaming is priced on capacity, throughput or both, and every model adds charges for storage and network transfer. Read each price list for what it counts: broker hours, partitions, gigabytes written and read, or units of elastic capacity.
Free entry points exist, with limits that matter in production. Confluent lists Basic starting at $0/month. Confluent offers $400 in free credits to new Confluent Cloud sign-ups to test its Basic, Standard, Enterprise and Freight cluster types. Google Cloud states that every calendar month, the first 10 GiB of throughput identified as the Message Delivery Basic SKU for a billing account is free, and after that the price is $40 per TiB in all Google Cloud regions. Redpanda offers a free start with no credit card required. A free tier is a way to test client compatibility, not a forecast of the production bill.
Serverless and per-gigabyte models charge for use rather than capacity. AWS states that with MSK Serverless you pay an hourly rate for your serverless clusters and an hourly rate for each partition that you create, plus per GB of data that your producers write and your consumers read. AWS lists Kinesis Data Streams pricing starting as low as $0.032 per GB of data ingested and $0.016 per GB of data retrieved. Partition charges punish designs with many small topics, so count partitions before you compare.
Tiers gate features as well as capacity. Microsoft advises choosing Basic for simple, low-volume streaming where you need a single consumer group and short retention. Kafka support, long retention, private networking and customer-managed keys can sit in higher tiers. Check which tier includes each feature on your list before comparing prices.
Self-managed Kafka has no license fee, and its cost is the cloud or hardware bill plus the engineers who run it. Price the people: on-call rotation, upgrades, partition rebalancing and recovery drills. A bring-your-own-cloud service moves the operations to the vendor and leaves the infrastructure bill with you.
| Meter | You are charged for | Grows with | Where it goes wrong |
|---|---|---|---|
| Broker or capacity hours | Provisioned brokers or capacity units, by the hour | Peak throughput and partition count | Paying for peak capacity all month |
| Throughput | Gigabytes written and read | Event volume and number of consumers | Fan-out to many consumers multiplies the read charge |
| Partitions | Each partition, by the hour | Topic design | Many small topics, each with many partitions |
| Storage | Gigabytes kept, by the month | Retention period and replication factor | Long retention on broker disks instead of tiered storage |
| Network transfer | Data crossing zones, regions or the internet | Replication and consumer location | Cross-zone replication left out of the estimate |
| Connectors and processing | Connector throughput and processing units | Pipelines and queries added over time | Billed separately from the broker and missed at renewal |
How long does implementation take for Data Streaming Platforms & Event Stream Processing?
A streaming rollout is a platform migration. Producers and consumers move in waves, and the old and new clusters run side by side until the last consumer has moved.
List every topic, its throughput, retention, partition count, producers and consumers. Record client libraries and versions, connectors and stream processing jobs.
Run a copy of production traffic on each shortlisted service for at least a week. Measure latency at the 99th percentile, test failover, and record the full bill.
Replicate topics from the old cluster with offsets preserved. Move consumers first, then producers, one application at a time, with a rollback plan for each.
Move stream processing jobs, register schemas, set access control per topic and retire the old cluster once every consumer has moved.
What should you ask vendors about Data Streaming Platforms & Event Stream Processing?
Ask each vendor to answer in writing and to demonstrate on your own throughput, retention and clients.
-
Do your producers and consumers already use Kafka clients?Yes Shortlist services that run Apache Kafka or accept the Kafka protocol, and test your own clients against each.No Include cloud-native messaging services and compare them on delivery model, ordering and retention.
-
Must the data stay inside your own cloud account?Yes Compare bring-your-own-cloud services with self-managed software, and count the infrastructure bill you will pay directly.No Compare managed services on throughput price, network charges and the tier that includes your required features.
-
Do you need joins, aggregations or real-time decisions on the stream?Yes Evaluate a stream processing engine alongside the platform, and test it on a stateful job.No Check that connectors and topic-to-table features cover delivery to your analytics systems.