EventStoreDB 20.6.0 Release

We are pleased to announce the official release of EventStoreDB 20.6.0! Thank you to everyone who tried out the previews and release candidates and provided us with some great feedback.

It is available for the following operating systems:

• Windows
• Ubuntu 16.04
• Ubuntu 18.04
• CentOS 7 (Commercial version)
• Amazon Linux 2 (Commercial version)
• Oracle Linux 7 (Commercial version)

Where can I get the packages?

Downloads are available on our website.

The packages can be installed using the following instructions.

Ubuntu 16.04/18.04 (via packagecloud)

curl -s https://packagecloud.io/install/repositories/EventStore/EventStore-OSS/script.deb.sh | sudo bash

sudo apt-get install eventstore-oss=20.6.0-1

Windows (via Chocolatey)

choco install eventstore-oss -version 20.6.0

.NET gRPC Client Packages

gRPC Streams Client

dotnet add package EventStore.Client.Grpc.Streams --version 20.6.0

gRPC Projection Management Client

dotnet add package EventStore.Client.Grpc.ProjectionManagement --version 20.6.0

gRPC Persistent Subscriptions Client

dotnet add package EventStore.Client.Grpc.PersistentSubscriptions --version 20.6.0

gRPC Operations Client

dotnet add package EventStore.Client.Grpc.Operations --version 20.6.0

gRPC User Management Client

dotnet add package EventStore.Client.Grpc.UserManagement --version 20.6.0

.NET TCP Client Packages

EventStore TCP Client

dotnet add package EventStore.Client --version 20.6.0

EventStore Embedded Client 

dotnet add package EventStore.Client.Embedded --version 20.6.0

Upgrade Procedure

Due to changes in the replication protocols and the way nodes gossip and host elections, there is no way to perform a rolling upgrade between version 5.x and version 20.6.0.

As such, the upgrade process would be as follows:

1. Take down the cluster
2. Perform an in-place upgrade of the nodes, ensuring that the relevant configuration and certificates are set up
3. Bring the nodes back online and wait for them to stabilise

Existing TCP clients

EventStoreDB v20.6.0 is compatible with the TCP client v5.0.8 and higher. Please note that the v5.0.8 client will only work with valid certificates if the client is cluster aware (i.e. using gossip to determine the best node to connect to).

Since TCP is disabled on v20.6.0 by default, you will need to enable this on the server. You can do this with the --enable-external-tcp configuration option.

If you are not using secured TCP connections, you will also need to disable external TLS on the server. This is set with --disable-external-tcp-tls.

If the client is cluster-aware and uses the cluster gossip to connect, it will need to have TLS terminated endpoints enabled in the cluster settings.

For example:

var clusterSettings = ClusterSettings.Create()
    .DiscoverClusterViaGossipSeeds()
    .SetGossipSeedEndPoints(tlsTerminatedEndpoints: true, gossipSeeds: gossipSeeds);

Existing Http clients

AtomPub is disabled by default as of 20.6.0. It can be enabled with the --enable-atom-pub-over-http setting.

The server also only supports HTTPS from version 20.6.0, so this will need to be taken into consideration.

Due to the changes to node role names, cluster-aware HTTP clients may need to update the cluster role definitions from Master/Slave to Leader/Follower.

Setting Up EventStoreDB

Development mode

Currently the easiest way to run EventStoreDB is to run in Development mode. You can do this by running EventStoreDB with the --dev flag.

This configures EventStoreDB to:

1. Use an existing development certificate. This means that when running in development mode, you don’t need to specify a certificate at startup.
2. Run in memory. By default, development mode will not write any data to disk. You can change this behaviour by overriding the MemDb setting.

Configuring a Cluster to run with Certificates

If you would like to run EventStoreDB without development mode, you will need to generate your own certificates. There are some requirements for these, especially if you intend on running a cluster of nodes.

These requirements are:

1. At least one IP address or DNS name needs to be specified in the SAN. The specified IP addresses or DNS names must match the IP/host of the EventStoreDB node.
2. The certificate’s subject needs to be CN=eventstoredb-node.
3. You will also need to configure EEventStoreDB with the path to the CA used to create these certificates.

EventStoreDB has a set of terraform scripts that generate the development CA and certificate. You can use these as a base for creating your own certificates.

Commercial customers will have access to the es-cli, which will include a command to generate these certificates for you.

Once you have the certificates generated, you will need to configure your nodes to use them. You can do this by adding the following configuration to any existing configuration:

TrustedRootCertificatesPath: {Path to CA Directory}
CertificateFile: {Path to Certificate File}
CertificatePrivateKeyFile: {Path to Private Key File} # Optional
CertificatePassword: changeit # Optional

Connecting a gRPC client

You will need to either trust the certificate authority you are running EventStoreDB with, or disable remote certificate validation on your client.

For testing or development with the .NET gRPC client, you can disable certificate validation in the client settings as follows:

var settings = new EventStoreClientSettings
{
    CreateHttpMessageHandler = () => new SocketsHttpHandler {
        SslOptions = {
            RemoteCertificateValidationCallback = (sender, certificate, chain, errors) => true,
        }
    },
};

Breaking Changes

These are the main breaking changes that will require changes to configuration or client code.

Cluster Role name change

The terminology used for the cluster node roles has been changed from Master and Slave to Leader and Follower respectively.

Following this change, there are a few things to take into consideration:

• The ES-RequireMaster header for HTTP will be deprecated in a future release and ES-RequireLeader should be used going forwards
• Monitoring scripts that leverage the gossip protocol will need to be updated to the new terminology

HTTP Prefixes removed

HTTP prefixes were dropped with the move from HTTPListener to Kestrel. Please be aware that EventStoreDB will fail if you try to specify HTTP Prefixes in the configuration.

HTTP Interface now uses HTTPS

The external HTTP interface has been changed to use HTTPS only. This means that a TLS certificate must be provided at startup to secure the interface.

This also means that external gossip is now over HTTPS. This may affect any cluster health status monitoring you have in place, as well as TCP Clients connecting to the cluster.

The TCP client has been updated to gossip over HTTPS by default. If you are using an older client, you will need to manually change it to use HTTPS when gossiping.

The process for setting up the required certificates is described in these release notes.

Tcp Client has TLS enabled by default

With EventStoreDB 20.6.0 we wanted to move to a secure by default approach and therefore the Tcp Client has been changed to have Tls Enabled by default.

Removal of undocumented projection selectors

Some projection selectors and options were removed:

• fromStreamCatalog selector
• fromStreamsMatching selector
• disableParallelism option
• catalogTransform option

These selectors were only usable in queries or transient projections. If you were using them, you need to recreate any affected queries or transient projections.

As only fromStreamsMatching was officially documented, it was the only one of these officially supported.

Usage of fromStreamsMatching can be replaced with fromAll with an appropriate where modifier.

Mutually exclusive security on TCP

Each TCP interface can be started as either secure or non secured, but not both at the same time. This can be controlled with the --disable-internal-tcp-tls and --disable-external-tcp-tls options. TLS is enabled for both interfaces by default.

Combined internal and external http interfaces

We have combined the internal and external http interfaces and therefore the previous configuration options of --ext-http-port and --int-http-port has been combined into --http-port and --int-http-port-advertise-as and --ext-http-port-advertise-as into --http-port-advertise-as.

Logging infrastructure moved to Serilog

With the release of EventStoreDB 20.6.0, the logging system moved over to Serilog. This does mean that the previous logging configuration is no longer valid for users who are upgrading.

At the time of release, the only configurable options in the provided logconfig.json are the log levels for various logging outputs.

Deprecated Features

These are features that are still available, but are flagged to be removed in the next major release.

Deprecation of Atom and TCP

As we move over to gRPC we are deprecating AtomPub and TCP. Both of these are disabled by default and command line options have been added to the server to enable both TCP and AtomPub.

AtomPub over HTTP can be enabled with --enable-atom-pub-over-http.

The external TCP interface can be enabled with --enable-external-tcp.

Note: AtomPub is enabled with dev mode, as being able to browse streams is a useful development tool.

Deprecation of clone nodes

Following the introduction of Read-Only Replicas, Clones have been disabled by default.

The new default behaviour is for excess nodes in the cluster to be terminated upon joining.

If you need to make use of Clones nodes while transitioning to using read replicas, the old behaviour can be restored by setting the --unsafe-allow-surplus-nodes option on all nodes in the cluster.

Main Headline Features

Important notes for macOS and the use of Docker

Although macOS was never a supported server platform for EventStoreDB, we have historically shipped binaries for those developing on that platform. In this preview release we have completed the move to gRPC for both client access and gossip which enables TLS by default.

The gRPC implementation for .NET Core uses the system-native TLS library for the platform on which it is running. In the case of macOS server-side ALPN is not supported by Security.Framework and consequently the Core CLR.

As a result of this, we are no longer shipping binaries for macOS server until such time as this is rectified. We recommend that users on macOS use the Docker container build of the server instead.

The gRPC client will still be supported on Linux, Windows and macOS.

.NET Core 3.1

EventStoreDB 20.6.0 only runs on Core CLR 3.1 which means that support for both the .NET Framework and Mono have been dropped.

Much of the platform-specific code has been removed, and we benefit from the focus on performance that Microsoft has given the Core CLR. We now use the Kestrel HTTP server, which has significant benefits over HttpListener which was used up to version 5.

Historically we have provided Linux and macOS binaries with the Mono runtime statically linked into them. Unfortunately this is not yet supported by .NET Core. Consequently we now ship the correct version of the runtime as part of our packages.

Introduction of gRPC

Since version 1.0 of EventStoreDB, two client protocols have been supported - Atompub over HTTP, and full-duplex protocol-buffers-over-TCP.

Each of these has advantages - the Protocol Buffers API is better for latency-sensitive workloads, while the Atom API gives more flexibility for cache design. However, both require a lot of work on each platform to implement.

To improve this situation, we have added a new default client protocol in EventStoreDB 20.6.0, using gRPC - a widely adopted standard which is interoperable across lots of platforms with minimal effort.

With EventStoreDB 20.6.0, we are shipping a .NET SDK for the new gRPC protocol.

Gossip over gRPC

With the preview releases of EventStoreDB 20.6.0, we’ve had great feedback from community members building out gRPC Clients and with that requested that we expose the cluster gossip protocol over gRPC for cluster aware clients to use.

The HTTP based gossip has not been removed and any existing operational tools and clients such as the EventStoreDB .NET TCP Client will continue to work, however due to the role name change, the consumers of the protocol will have to be updated to use Follower instead of Slave and Leader instead of Master.

The EventStoreDB .NET TCP Client 5.x has been updated in 5.0.8 to understand the new cluster terminology.

Server Side Filtering

We have also added the ability to filter both reads and subscriptions on the server-side. You can filter by either event type or stream name.

The intention here is to remove the need for having to use projections when you only want to subscribe to EventStoreDB and only receive events from streams or events satisfying a particular filter.

New methods and overloads have been added for reads and subscriptions to the .NET TCP Client SDK.

New HTTP endpoints have been added as well for reading only.

The gRPC Client SDK only contains support for filtered subscriptions.

Thanks to @Yspadadden for providing the inspiration for this feature.

Read-Only Replicas

A common use case for adding more nodes to a cluster is being able to provide data from EventStoreDB closer to the destination where it’s intended to be used. However, by adding more nodes to the cluster, you incur write latency as EventStoreDB requires a majority of the nodes in the cluster to acknowledge the write before it’s deemed successful.

Previously EventStoreDB supported adding additional nodes to the cluster as clones (which as of EventStoreDB 20.6.0 has been deprecated) for scaling out reads.

This has led to a few issues as clones can be promoted into the cluster which can mean that clients end up reading from a node that starts participating in normal quorum operations. This has two undesired side effects, in that additional load can be inadvertently placed on a quorum node, and in the event of network segregation the cluster can end up in a split brain situation.

To address both of these concerns, we have introduced the ability to mark a node as a read-only replica which avoids these scenarios. This type of node will not participate in elections, and will not be promotable to a clone, follower or leader node.

To start a node as a read-only replica, the ReadOnlyReplica option has been introduced.

%> EventStore.ClusterNode.exe --read-only-replica

Both the EventStoreDB TCP and gRPC Clients have also been adjusted to allow the user to set the node preference to ReadOnlyReplica.

Leader Resignation

Performing maintenance work such as scavenging on a node can result in performance degradation.

For example, scavenging is an I/O intensive operation, and running a scavenge on a leader node can affect the node’s ability to serve write requests.

Providing the ability to resign the leader node gives customers the ability to perform maintenance work on a node without having to shutdown or remove the node from the cluster.

Resigning the leader node is currently a two-step process which leverages a previously available startup time feature called node priority.

Two new admin HTTP endpoints have been added, priority to allow setting the node priority and resign to allow resigning the node.

To resign a leader node, the user has to issue two HTTP POST requests.

The first action is to reduce the current leader node’s priority so that during elections, another node with a higher priority will be chosen over the current leader node.

%> curl -X POST -d {} https://localhost:2113/admin/node/priority/-1 -u admin:changeit

The second action is to issue a resignation command which will explicitly start a round of elections.

%> curl -X POST -d {} https://localhost:2113/admin/node/resign -u admin:changeit

The above commands are privileged commands and require either a user in the $ops or $admins group to perform.

Restarting the Projection Subsystem

There are a few situations where the projection subsystem can get stuck and can often only be recovered by restarting the leader node. Restarting a node is not ideal as this interrupts read and write operations, and the node may take a while to start up again.

A new operation which allows you to just restart the projection subsystem has been added. This will stop and start the subsystem as if the node had been shut down, but without interrupting any other operations on the node. During the restart, the projections may enter a “suspended” state as they dispose of their subscriptions.

To restart the projection subsystem, a user has to issue an HTTP POST request to the new endpoint.

%> curl -X POST -d {} https://localhost:2113/projections/restart -u admin:changeit

The above command is a privileged command and requires either a user in the $ops or $admins group to perform.

Pinned by Correlation Id Strategy for Persistent Subscriptions

A new competing consumer strategy has been added to persistent subscriptions, which presents events to the same consumer if they have the same correlation id.

In order to use this new strategy, the $by_correlation_id projection must be running, and you can now select this new strategy when creating the subscription through the UI or TCP client.

Thanks to @sammosampson for this feature!

Liveness HealthCheck

We have introduced a health check which can be queried via the {server_address}/health/live endpoint. This endpoint will return a 204 status code once EventStoreDB is ready to serve requests.

It is configured as the default health check in the Docker container.

Internal Security in a cluster

The internal interfaces in a cluster are now secured by default and will no longer be externally accessible.

Internal TCP connections are mutually authenticated, and both follower and leader nodes will verify the certificates of the other nodes when a connection is made. This behaviour can be disabled using the --disable-internal-tcp-tls option, and is disabled by default in development mode (--dev).

In addition to the certificate requirement, the subject of the certificate (CN=eventstoredb-node) is also used to authorize a node as a system user.

New options for configuration of certificates

To make configuration easier EventStoreDB now supports multiple options for configuring certificates to allow you to choose what best suits your setup. The options provided have also been updated to have more detailed descriptions and errors to help guide customers during setup.

A certificate can be provided from :

• A PKCS #12 (.p12/.pfx) certificate file, by providing it to the --certificate-file option
• A public and private key pair, by setting --certificate-file to point to an X.509 (.pem, .crt, .cer, .der) certificate file, and pointing --certificate-private-key-file to a private key (.key)
• The windows certificate store, using the --certificate-store-location, --certificate-store-name, --certificate-thumbprint and --certificate-subject-name options

Max Truncation Safety Feature (Released in the Release Candidate)

We have added a feature to avoid large unexpected truncations by adding a MaxTruncation setting that will stop any truncations over that size. This can be set via the following command-line argument, and can be disabled by setting it to -1.

--max-truncation=-1

The default is 1 chunk of data.

Thanks to @megakid for this feature.

Ability to use Host names in gossip and advertise endpoints (Released in the Release Candidate)

In the past EventStoreDB has only supported using IP addresses in the gossip and advertise endpoints. We have now provided the ability to use hostname and port in the gossip seeds as well as the advertise endpoints.

Projection Improvements

A number of improvements have been made to projections with a focus on stability and bug fixes.

This includes:

• Exponential backoff with jitter when retrying writes
• A projection will retry more times before faulting
• Decreasing the number of writes projections perform
• Bug fixes, which can be found in the changelogs

Cluster Improvements

We have worked on improving cluster stability by reducing the number of elections triggered by normal operations:

• Nodes that have just started up will first try and identify and subscribe to an existing leader instead of triggering elections. This can be helpful during maintenance and upgrades
• When a node loses TCP connection to the leader, it will issue a gossip and determine whether the leader is dead or not based on if it gets a response
• Information about the epoch has been added to election messages to allow nodes to make better decisions during elections

This should result in fewer leader change overs and fewer offline truncations.

We have also updated the cluster replication to ensure that dirty reads from followers are no longer possible.

Documentation

The documentation for version 20.6.0 has not been completed yet, but it is actively being worked on.

We are also addressing many issues with the existing documentation and working to improve its overall quality.

If you have any questions that aren’t covered in these release notes, please feel free to reach out on discuss, github or slack.

Providing Feedback

If you encounter any issues, please don’t hesitate to open an issue on GitHub if there isn’t one already.

Additionally, there is a fairly active discuss channel, and an #eventstore channel on the DDD-CQRS-ES slack community.