Introduction
I start to write this article to know more about message brokers and Event Streaming Platforms, and to understand more about how Event-Driven applications work.
Message brokers and event streaming platforms are two important technologies that are used to decouple applications and enable asynchronous communication. Message brokers are a more traditional technology that stores messages in queues until they are consumed by applications. Event streaming platforms are a newer technology that stores events in a continuous stream, which allows applications to process events as they happen.
Both message brokers and event streaming platforms have their advantages and disadvantages. Message brokers are typically easier to set up and use, but they can be less scalable and performant than event streaming platforms. Event streaming platforms are more scalable and performant, but they can be more complex to set up and use.
The choice of whether to use a message broker or an event streaming platform depends on the specific requirements of the application. If the application needs to be able to process a large volume of events in real-time, then an event streaming platform is a good choice. If the application needs to be simple to set up and use, then a message broker is a good choice.
Event-Driven Architecture
As this article in Amazon explains that event-driven architecture uses events to trigger and communicate between decoupled services and is common in modern applications built with microservices. An event is a state change, or an update, like an item being placed in a shopping cart on an e-commerce website. Events can either carry the state (the item purchased, its price, and a delivery address) or events can be identifiers (a notification that an order was shipped).
Event-driven architectures have three key components: event producers, event routers, and event consumers. A producer publishes an event to the router, which filters and pushes the events to consumers. Producer services and consumer services are decoupled, which allows them to be scaled, updated, and deployed independently.
This article on the RedHat website, explains that Event-driven architecture is made up of event producers and event consumers. An event producer detects or senses an event and represents the event as a message. It does not know the consumer of the event or the outcome of an event.
After an event has been detected, it is transmitted from the event producer to the event consumers through event channels, where an event processing platform processes the event asynchronously
Also, says that event-driven architecture may be based on either a pub/sub model or an event stream model.
Pub/sub model
This is a messaging infrastructure based on subscriptions to an event stream. With this model, after an event occurs, or is published, it is sent to subscribers that need to be informed.
Event streaming model
With an event streaming model, events are written to a log. Event consumers don’t subscribe to an event stream. Instead, they can read from any part of the stream and can join the stream at any time.
What is a message broker?
A message broker is a software that enables applications, systems, and services to communicate with each other and exchange information. The message broker does this by translating messages between formal messaging protocols. This allows interdependent services to “talk” with one another directly, even if they were written in different languages or implemented on different platforms.
What is event streaming?
This article on the Apache Kafka website explains that event streaming is the practice of capturing data in real-time from event sources like databases, sensors, mobile devices, cloud services, and software applications in the form of streams of events; storing these event streams durably for later retrieval; manipulating, processing, and reacting to the event streams in real-time as well as retrospectively; and routing the event streams to different destination technologies as needed. Event streaming thus ensures a continuous flow and interpretation of data so that the right information is at the right place, at the right time.
What can I use event streaming for?
As this article says, event streaming is applied to many cases across a plethora of industries and organizations. Its many examples include:
- To process payments and financial transactions in real-time, such as in stock exchanges, banks, and insurance.
- To track and monitor cars, trucks, fleets, and shipments in real-time, such as in logistics and the automotive industry.
- To continuously capture and analyze sensor data from IoT devices or other equipment, such as in factories and wind parks.
- To collect and immediately react to customer interactions and orders, such as in retail, the hotel and travel industry, and mobile applications.
- To monitor patients in hospital care and predict changes in condition to ensure timely treatment in emergencies.
- To connect, store, and make available data produced by different divisions of a company.
- To serve as the foundation for data platforms, event-driven architectures, and microservices.
Difference between Message Brokers and Streaming Platforms
Whereas message brokers can support two or more messaging patterns, including message queues and pub/sub, event streaming platforms only offer pub/sub-style distribution patterns. Designed for use with high volumes of messages, event streaming platforms are readily scalable. They’re capable of ordering streams of records into categories called topics and storing them for a predetermined amount of time. Unlike message brokers, however, event streaming platforms cannot guarantee message delivery or track which consumers have received messages.
Event streaming platforms offer more scalability than message brokers but fewer features that ensure fault tolerance (like message resending), as well as more limited message routing and queueing capabilities.
List of Message Brokers and Streaming Platforms
Here is the list of message brokers and event-streaming platforms with their features. This is not a comprehensive list, many of the projects in this list are popular software that has been used for many years like Apache Kafka and RabbitMQ.
1. Apache Kafka
It is an open-source distributed streaming platform, which is known for its high throughput, scalability, and fault tolerance. Apache Kafka (Kafka) is an open-source, distributed streaming platform that enables (among other things) the development of real-time, event-driven applications. Kafka has three primary capabilities:
- It enables applications to publish or subscribe to data or event streams.
- It stores records accurately (i.e., in the order in which they occurred) in a fault-tolerant and durable way.
- It processes records in real-time (as they occur).
According to this article published by IBM, developers can leverage these Kafka capabilities through four APIs:
Producer API: This enables an application to publish a stream to a Kafka topic. A topic is a named log that stores the records in the order they occurred relative to one another. After a record is written to a topic, it can’t be altered or deleted; instead, it remains in the topic for a preconfigured amount of time—for example, for two days—or until storage space runs out.
Consumer API: This enables an application to subscribe to one or more topics and to ingest and process the stream stored in the topic. It can work with records in the topic in real time, or it can ingest and process past records.
Streams API: This builds on the Producer and Consumer APIs and adds complex processing capabilities that enable an application to perform continuous, front-to-back stream processing specifically, to consume records from one or more topics, to analyze or aggregate or transform them as required, and to publish resulting streams to the same topics or other topics. While the Producer and Consumer APIs can be used for simple stream processing, it’s the Streams API that enables the development of more sophisticated data- and event-streaming applications.
Connector API: This lets developers build connectors, which are reusable producers or consumers that simplify and automate the integration of a data source into a Kafka cluster. Check out some ready-made connectors for popular data stores.
2. RabbitMQ
RabbitMQ is a messaging broker - an intermediary for messaging. It gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
Feature highlights
Reliability: RabbitMQ offers a variety of features to let you trade off performance with reliability, including persistence, delivery acknowledgments, publisher confirms, and high availability.
Flexible routing: Messages are routed through exchanges before arriving at queues. RabbitMQ features several built-in exchange types for the typical routing logic. For more complex routing you can bind exchanges together or even write your own exchange type as a plugin.
Clustering: Several RabbitMQ servers on a local network can be clustered together, forming a single logical broker.
Federation: For servers that need to be more loosely and unreliably connected than clustering allows, RabbitMQ offers a federation model.
Highly available queues: RabbitMQ supports replicated queues and streams. machines in a cluster, ensuring that even in the event of hardware failure, messages are safe as long as there is a majority of cluster nodes online.
Multi-protocol: RabbitMQ supports messaging over a variety of messaging protocols.
Many clients: There are RabbitMQ clients in almost any language you can think of.
Management UI: RabbitMQ ships with an easy-to-use management UI that allows you to monitor and control every aspect of your message broker.
Tracing: If your messaging system is misbehaving, RabbitMQ offers tracing support to let you find out what's going on.
Plugin system: RabbitMQ ships with a variety of plugins extending it in different ways, and you can also write your own.
3. ActiveMQ
Apache ActiveMQ® is the most popular open-source, multi-protocol, Java-based message broker. It supports industry-standard protocols so users get the benefits of client choices across a broad range of languages and platforms. Connect with clients written in JavaScript, C, C++, Python, .Net, and more. Integrate your multi-platform applications using the ubiquitous AMQP protocol. Exchange messages between your web applications using STOMP over WebSockets. Manage your IoT devices using MQTT. Support your existing JMS infrastructure and beyond. ActiveMQ offers the power and flexibility to support any messaging use case.
Features
- Supports a variety of Cross Language Clients and Protocols from Java, C, C++, C#, Ruby, Perl, Python, PHP
- OpenWire for high-performance clients in Java, C, C++, C#
- Stomp support so that clients can be written easily in C, Ruby, Perl, Python, PHP, ActionScript/Flash, Smalltalk to talk to ActiveMQ as well as any other popular Message Broker
- AMQP v1.0 support
- MQTT v3.1 support allowing for connections in an IoT environment.
- full support for the Enterprise Integration Patterns both in the JMS client and the Message Broker
- Supports many advanced features such as Message Groups, Virtual Destinations, Wildcards and Composite Destinations
- Fully supports JMS 1.1 & 2.0 and J2EE 1.4+ with support for transient, persistent, transactional and XA messaging
- Spring Support so that ActiveMQ can be easily embedded into Spring applications and configured using Spring’s XML configuration mechanism
- Tested inside popular J2EE servers such as TomEE, Geronimo, JBoss, GlassFish and WebLogic
- Includes JCA 1.5 resource adaptors for inbound & outbound messaging so that ActiveMQ should auto-deploy in any J2EE 1.4 compliant server
- Supports pluggable transport protocols such as in-VM, TCP, SSL, NIO, UDP, multicast, JGroups and JXTA transports
- Supports very fast persistence using JDBC along with a high performance journal
- Designed for high performance clustering, client-server, peer based communication
- REST API to provide technology agnostic and language neutral web based API to messaging
- Ajax to support web streaming support to web browsers using pure DHTML, allowing web browsers to be part of the messaging fabric
- CXF and Axis Support so that ActiveMQ can be easily dropped into either of these web service stacks to provide reliable messaging
- Can be used as an in-memory JMS provider, ideal for unit testing JMS
4. Mosquitto
It is an open-source message broker that implements the MQTT (Message Queuing Telemetry Transport) protocol, which is used for IoT (Internet of Things) applications.
Eclipse Mosquitto implements the MQTT protocol versions 5.0, 3.1.1 and 3.1. Mosquitto is lightweight and is suitable for use on all devices from low-power single-board computers to full servers.
The MQTT protocol provides a lightweight method of carrying out messaging using a publish/subscribe model. This makes it suitable for Internet of Things messaging such as with low power sensors or mobile devices such as phones, embedded computers or microcontrollers.
The Mosquitto project also provides a C library for implementing MQTT clients and the very popular mosquitto_pub and mosquitto_sub command line MQTT clients.
5. ZeroMQ
ZeroMQ (also spelled ØMQ, 0MQ or ZMQ) is a high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker.
ZeroMQ supports common messaging patterns (pub/sub, request/reply, client/server and others) over a variety of transports (TCP, in-process, inter-process, multicast, WebSocket and more), making inter-process messaging as simple as inter-thread messaging. This keeps your code clear, modular and extremely easy to scale.
ZeroMQ is developed by a large community of contributors. There are third-party bindings for many popular programming languages and native ports for C# and Java.
6. Apache Pulsar
Pulsar is a multi-tenant, high-performance solution for server-to-server messaging. Originally developed by Yahoo, Pulsar is under the stewardship of the Apache Software Foundation.
Key features of Pulsar are listed below:
- Native support for multiple clusters in a Pulsar instance, with seamless geo-replication of messages across clusters.
- Very low publish and end-to-end latency.
- Seamless scalability to over a million topics.
- A simple client API with bindings for Java, Go, Python and C++.
- Multiple subscription types (exclusive, shared, and failover) for topics.
- Guaranteed message delivery with persistent message storage provided by Apache BookKeeper. A serverless lightweight computing framework Pulsar Functions offers the capability for stream-native data processing.
- A serverless connector framework Pulsar IO, which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar.
- Tiered Storage offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out.
7. NATS
NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Foundation (CNCF). NATS has over 40 client language implementations, and its server can run on-premise, in the cloud, at the edge, and even on a Raspberry Pi. NATS can secure and simplify the design and operation of modern distributed systems.
Use cases
NATS can run anywhere, from large servers and cloud instances, through edge gateways and even IoT devices. Use cases for NATS include:
- Cloud Messaging
- Services (microservices, service mesh)
- Event/Data Streaming (observability, analytics, ML/AI)
- Command and Control
- IoT and Edge
- Telemetry / Sensor Data / Command and Control
- Augmenting or Replacing Legacy Messaging Systems
8. Apache RocketMQ
Apache RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability.
It offers a variety of features:
- Messaging patterns including publish/subscribe, request/reply and streaming
- Financial grade transactional message
- Built-in fault tolerance and high availability configuration options based on DLedger Controller
- Built-in message tracing capability also supports open-tracing
- Versatile big-data and streaming ecosystem integration
- Message retroactivity by time or offset
- Reliable FIFO and strictly ordered messaging in the same queue
- Efficient pull and push consumption model
- Million-level message accumulation capacity in a single queue
- Multiple messaging protocols like gRPC, MQTT, JMS and OpenMessaging
- Flexible distributed scale-out deployment architecture
- Lightning-fast batch message exchange system
- Various message filter mechanics such as SQL and Tag
- Docker images for isolated testing and cloud-isolated clusters
- Feature-rich administrative dashboard for configuration, metrics and monitoring
- Authentication and authorization
- Free open-source connectors, for both sources and sinks
- Lightweight real-time computing
9. EMQ X
EMQX broker with a high-performance real-time message processing engine, powering event streaming for IoT devices at a massive scale.
As the most scalable MQTT broker, EMQX can help you connect any device, at any scale. Move and process your IoT data anywhere.
Benefits
- Massive scale: Scale to 100 million concurrent MQTT connections with a single EMQX 5.0 cluster.
- High performance: Move and process millions of MQTT messages per second in a single broker.
- Low latency: Guarantee sub-millisecond latency in message delivery with soft real-time runtime.
- Fully MQTT 5.0: 100% compliant with MQTT 5.0 and 3. x standard for better scalability, security, and reliability.
- High availability: Achieve high availability and horizontal scalability through a masterless distributed architecture.
- Cloud-native & K8s: Easy to deploy on-premises or in public clouds with Kubernetes Operator and Terraform.
Conclusion
Message brokers and event streaming platforms are both powerful technologies that can be used to decouple applications and enable asynchronous communication. The choice of which technology to use depends on the specific requirements of the application.
In conclusion, message brokers and event streaming platforms are both important technologies that can be used to build modern, scalable, and event-driven applications. By understanding the differences between these two technologies, you can make the best choice for your specific needs.
Resources
Apache Pulsar documentation
NATS documentation
Apache Kafka Introduction
ZeroMQ documentation
RabbitMQ documentation
EMQX documentation
Apache RocketMQ documentation
What is Apache Kafka?
ActiveMQ documentation
Mosquitto documentation
What are Message Brokers?