June is here and, as planned, EventStore 21.6 has been shipped. It is a substantial release containing important usability, stability, performance, and fixes and features as well as being an important step towards EventStoreDB's strategic architectural vision.
A full set of release notes can be found here but I wanted to take you through some of the core features of this release in more detail.
Persistent subscription: subscribe to all
One of the most frustrating limitations of EventStoreDB is that you were never able to create a persistent subscription that subscribed to the $all stream. This often led to our users using the following work around:
They would first have to set up a projection that subscribed to the $all stream and project the whole thing into a new stream which they could then set up the persistent subscription to. This had some obvious downsides, such as duplicating the number of events in the store as well as adding latency between the event being written and then being picked up by a subscriber.
This both significantly simplifies this pattern and improves performance.
Improvements to gRPC appends
gRPC has been EventStoreDB’s chosen client-server protocol since the 20.6 release and we continue to make improvements to its performance with every release. With the latest version we have added the ability to use multiplex appends over a single gRPC channel. This improves speed by eliminating the channel setup overhead. In order to take advantage of this improvement appends need to be made with the default credentials for the connection because credentials are passed during channel setup. This results in a performance improvement in the order of 20x.Note: If you need to set the credentials on a per call basis, this is still supported but won’t take advantage of the improvements to append speed.
The new projection engine
For a long time now EventStoreDB has been using the v8 Javascript engine for its internal stream projections. This was a huge overhead when developing the product because of the build tool chain required to compile and manage it. To address this EventStoreDB has replaced the v8 Javascript engine with JInt.
This comes with benefits for EventStoreDB users. EventStoreDB will now run on ARM, opening up hosting options as well as speed improvements.
This change also sets up EventStoreDB for more improvements to stream projections in the future. For example:
- Ad Hoc queries to return a one time stream of events which will improve the user experience for our current one off querying capabilities.
- Better debugging experience, making it easier for you to build and test your projections.
- A broader programmability layer, including stored procedure-like functionality that will allow you to do things such as validating appends on the server.
- Other ways to index streams beyond the default stream index: for example, you will be able to index your events by time allowing you to do time based queries.
These improvements and more are planned for stream projections, one of the integral features of EventStoreDB.
Auto configuration on start up
Over the next year you will see us make a move away from required or default configuration options for EventStoreDB to automatically tuned options, making EventStoreDB easier to configure and host. We recognize that configuring EventStoreDB right now is complex and error prone, so we are consistently moving toward supporting dynamic, context aware configuration starting with this release.
The work on Event Store Cloud over the past 12 months has given us a better understanding of how EventStoreDB behaves in different cloud hosting scenarios. This has allowed us to automatically set some configuration options based upon the environment in which it is running. We've started with the following:
- StreamInfoCacheCapacity
- ReaderThreadCount
- WorkerThreads
You can find out more about these in the release notes and we will continue to add more over time.
Log version 3
Another substantial feature coming into EventStoreDB 21.6 is Log v3. There has been substantial work over the past year on a new storage engine and log format that represents a foundational step towards the next generation of EventStoreDB.
EventStoreDB has been around since 2012 and although it is already considered the leader in Event Store class databases this is the first shipping step in an ambitious strategic architectural plan.
For the 21.6 release Log V3 is not the default. Although it has gone through substantial testing it is foundational so we want to get through a hardening cycle. It will run in EventStoreDB continuous integration and we encourage users to test it on their workloads. It is anticipated that Log V3 will become the default format in the next release (22.2) and there will be an online migration path that we will discuss in more detail as we get closer to that release.
Log V3 can be enabled by setting the DbLogFormat option to ExperimentalV3`.
As it is, Log V3 will result in speed improvements but more importantly opens the door to a large number of strategic features. These include (but are not limited to):
- System and some user projections will be replaced with indexes which will substantially reduce write amplification and improve lookup times.
- Multiple logs (multi-log) which will introduce the concept of a database to EventStoreDB. This will allow you to manage multiple logs in a given cluster. This enables important features such as crypto-assured immutability, multi-tenancy, archiving, and further specialised logs to natively support GDPR and high ingress scenarios to be developed and supported in the future.
These features and more will be discussed in more depth in future content planned for later this summer.
Dave Remy, Event Store CEO, says:
EventStoreDB 21.6 release is important for Event Store as well as EventStoreDB users, it delivers substantial features and it delivers a foundational step in EventStoreDB’s larger strategic architectural plan. EventStoreDB is approaching a decade in production with workloads all over the world, across industries, and across tech stacks. Our ambition is to further grow EventStoreDB’s capability to hyperscale workloads taking advantage of the latest advances in database and cloud innovation. The foundation of any database is the underlying storage engine and file system format. Log V3 in 21.6 represents a rewrite of this foundation in a way that addresses this ambition while assuring stability and a migration path. Stay tuned, over the next few months, and years as these strategic architecture features roll out. Event Store is committed to being the leading operational, “source of truth” database for the modern enterprise.