EventStoreDB and Kafka are two popular technologies that can be used together to build a highly scalable and reliable event-driven architecture. Let’s go into more detail about what these technologies are and how exactly they complement each other.
What's the difference between EventStoreDB and Kafka?
EventStoreDB is an operational database, specifically designed for event sourcing. It allows you to store state transition as an immutable sequence of events. You can read events from a stream, subscribe to a stream to receive new events as they arrive, and append events to a stream.
Kafka, on the other hand, is a distributed data streaming technology. It allows you to publish messages to a topic, which is a stream of data, and subscribe to a topic to receive messages as they arrive. Kafka is highly scalable and can handle large volumes of data.
EventStoreDB is about storing data and streaming data both in real-time as well as replay. Kafka is about data in motion (streaming data).
How can EventStoreDB and Kafka work together?
In simple terms, imagine EventStoreDB as the brain, which holds all of your information, and picture Kafka as the nervous system. EventStoreDB can be used to store state transitions as events, while Kafka can be used to distribute the events in real-time. Watch this webinar to see why it's important to store events - not just stream them.
When used together, EventStoreDB and Kafka allow you to build a system that can handle a large volume of events, while still maintaining a high level of performance.
How can systems benefit from using EventStoreDB and Kafka?
One common example is for event driven applications composed of one or more microservices. Within the application, state transition events are stored as the underlying source of truth. Projections can be used to transform detailed events into to more meaningful events. Kafka is then used to distribute these events out to other systems and analytics technologies in the enterprise.
Another strategy, if an enterprise is not in position to rewrite part or all of an existing application into an event oriented application, is to use EventStoreDB as a "sidecar source of truth". Low level events from the existing application can be captured in EventStoreDB. Projections can be used to transform those low level events into more meaningful domain and integration events, which can then be distributed by an integration bus like Kafka.
This has the advantage of being able to process events in real time in a manner that Kafka was not built for. Also, a persisted record of the events can be important for explaining insights or errors that occur after the fact in downstream systems or data technologies.
Find out more about EventStoreDB.
Watch our webinar, Why EventStoreDB? Simplified, as Event Store's Stephen Tung will talk you through:
- What even is EventStoreDB?
- How the operational database can help you
- How to communicate the why to peers
- Use cases