Gossip Seed configuration and Certificate Management for EventStoreDB

Nodes in a cluster can discover the other nodes using one of two methods - gossip seeds, or DNS discovery.

This post describes how to configure the certificates and nodes for a cluster using gossip seed discovery in EventStoreDB versions 21.10 and beyond.

These instructions explain how to configure EventStoreDB with a wildcard certificate (from a public or private Certificate Authority (CA)) and gossip seed discovery, how to rotate the certificates, and how to replace a node with this setup.

This setup also assumes that the clients have the root Certificate Authority certificate installed.

Note: See this post for similar instructions for setting up DNS discovery

How Gossip Seed Discovery works

Each EventStoreDB node maintains a list of all of the nodes in its cluster. This list contains the addresses of the nodes as well as information such as the nodes’ roles, status, and how caught up they are.

The nodes periodically "gossip" with each other by calling the gossip endpoint on one of the other nodes. If this call succeeds, then the node updates its information based on new information from the other node. If it fails, then it will attempt to call a different node on the next gossip. There are further steps taken for possibly dead nodes, but that's beyond the scope of this post.

A Gossip Seed is a list of addresses that are provided to a node at startup to kick off this process. When starting up, the node selects an address from the gossip seed and attempts to gossip to it. Once the node has successfully gossiped with another node in the cluster, it will use the gossip information from the other node going forward rather than the gossip seed.

In this way, new nodes can be added to the cluster even if they are not present in the initial gossip seed. The new node just needs to have a gossip seed with the old node's addresses in it.

Initial Setup

The initial setup of EventStoreDB will be a 3-node cluster. All nodes will use *.project.example.com as a wildcard certificate example.

Each node has 1 IP address used both for internal (node to node) communication and client communication.

Node addresses

The following table lists the addresses of the nodes in our cluster:

Certificate Configuration

Public CA:

The certificate private key must be in RSA format. The certificate needs to have the following attributes:

Subject Name - Common Name (CN): *.project.example.com
Subject Alternative Name (SAN): *.project.example.com

Assuming that for each node the public key is in a file called node.crt and the private key in a file called node.key, copy the certificates to their respective nodes. The relevant configuration for the certificates for all nodes is:

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/ssl/certs
CertificateReservedNodeCommonName: "*.project.example.com"

Note: The TrustedRootCertificatesPath is the standard location of trusted root certificates in Linux-based systems and is used in this case since the root certificate is a public Certificate Authority (CA).

Private CA:

This assumes you are using The Event Store gencert-cli.

Generate the root CA

./es-gencert-cli create-ca

Generate each node certificates

./es-gencert-cli.exe create-node --dns-names *.project.example.com -out ./certs/n1
./es-gencert-cli.exe create-node --dns-names *.project.example.com -out ./certs/n2
./es-gencert-cli.exe create-node --dns-names *.project.example.com -out ./certs/n3

Copy each node.crt and node.key files to /etc/eventstore/certs on each respective node. Copy the public key of the root certificate authority, the ca.crt file, to /etc/eventstore/certs/ca on each node.

The relevant configuration for the certificates for each node is:

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

Note: The TrustedRootCertificatesPath in this case is not the standard location of trusted root certificates in Linux-based systems.

Node Configuration

Set the IntHostAdvertiseAs and ExtHostAdvertiseAs settings to the hostname of the node, and the GossipSeed to the hostnames of the other nodes in the cluster. This gives you a network configuration section like this:

# Network configuration
IntIp: 127.0.20.1
ExtIp: 127.0.20.1
IntHostAdvertiseAs: node1.project.example.com
ExtHostAdvertiseAs: node1.project.example.com

# Cluster gossip
ClusterSize: 3
DiscoverViaDns: false
GossipSeed: node2.project.example.com:2113,node3.project.example.com:2113

In brief, these settings do the following:

• IntIp - The IP address used by internal TCP connections between nodes in a cluster. This is solely used for replication.
• ExtIp - The IP address used for client TCP connections, and for all HTTP and gRPC communication. This includes gossip and elections between nodes.
• IntHostAdvertiseAs - The hostname used by internal TCP connections between nodes in a cluster. This is solely used for replication.
• ExtHostAdvertiseAs - The hostname used for client TCP connections, and for all HTTP and gRPC communication. This includes gossip and elections between nodes.
• ClusterSize - number of nodes in the cluster. This excludes non-promotable nodes such as ReadOnlyReplicas.
• DiscoverViaDns - the other method of discovery in a cluster. See this other article for more information. 
• GossipSeed - the other nodes in the cluster, which this node will try to communicate with. This list excludes the current node.

Note: If you are seeing duplicate nodes in the gossip for a cluster, ensure that the GossipSeeds on the nodes correspond with the ExtHostAdvertiseAs entries on the other nodes.

Each node configuration is then, in the case of private CA: (for public CA issued certificates, replace the certificate configuration section)

---
# Paths
Db: /var/lib/eventstore
Index: /var/lib/eventstore/index
Log: /var/log /eventstore

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

# Network configuration
IntIp: 127.0.20.1
ExtIp: 127.0.20.1
IntHostAdvertiseAs: node1.project.example.com
ExtHostAdvertiseAs: node1.project.example.com
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true

# Cluster gossip
ClusterSize: 3
DiscoverViaDns: false
GossipSeed: node2.project.example.com:2113,node3.project.example.com:2113

# Projections configuration
RunProjections: All

---
# Paths
Db: /var/lib/eventstore
Index: /var/lib/eventstore/index
Log: /var/log /eventstore

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

# Network configuration
IntIp: 127.0.20.2
ExtIp: 127.0.20.2
IntHostAdvertiseAs: node2.project.example.com
ExtHostAdvertiseAs: node2.project.example.com
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true

# Cluster gossip
ClusterSize: 3
DiscoverViaDns: false
GossipSeed: node1.project.example.com:2113,node3.project.example.com:2113

# Projections configuration
RunProjections: All

---
# Paths
Db: /var/lib/eventstore
Index: /var/lib/eventstore/index
Log: /var/log /eventstore

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

# Network configuration
IntIp: 127.0.20.3
ExtIp: 127.0.20.3
IntHostAdvertiseAs: node3.project.example.com
ExtHostAdvertiseAs: node3.project.example.com
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true

# Cluster gossip
ClusterSize: 3
DiscoverViaDns: false
GossipSeed: node1.project.example.com:2113,node2.project.example.com:2113

# Projections configuration
RunProjections: All

Client connection string

You can provide the gossip seed as a comma-separated list in the connection string.

When trying to establish a connection, the client will call the addresses in the gossip seed. Once it gets a response, it will use the returned information to find the best node to connect to based on its status and role.

Note that when using certificates signed by a private CA, the root CA public key must be trusted by the client. Typically, it is placed in the trusted store of the user under which the application is running.

gRPC

esdb://node1.project.example.com:2113,node2.project.example.com:2113,node3.project.example.com:2113

TCP

GossipSeeds=node1.project.example.com:2113,node2.project.example.com:2113,node3.project.example.com:2113;

Alternatively, if you set up a DNS server that returns the node's addresses in a round-robin fashion, then you could provide just that address to your clients. For example:

Cluster DNS Entry

Create the following DNS records for your cluster:

cluster.project.example.com. 300 IN A 127.0.20.1
cluster.project.example.com. 300 IN A 127.0.20.2
cluster.project.example.com. 300 IN A 127.0.20.3

gRPC using Cluster DNS

esdb+discover://cluster.project.example.com:2113

TCP using Cluster DNS

GossipSeeds=cluster.project.example.com:2113;

Advertising a different hostname to clients

You may want to configure your cluster so that nodes gossip to clients using a different address to the other nodes in the cluster:

To do this, you can follow the steps above with the following changes:

Set AdvertiseHostToClientAs in node configuration

Add the `AdvertiseHostToClientAs` option to your node's network configuration, and set it to the external hostname.

This changes the address that the node advertises to clients that gossip with it, but does not affect the address that the nodes use to gossip internally.

All other configuration options remain the exact same. For example:

# Network configuration
IntIp: 127.0.20.1
ExtIp: 127.0.20.1
IntHostAdvertiseAs: node1.project.example.com
ExtHostAdvertiseAs: node1.project.example.com
AdvertiseHostToClientAs: public1.project.example.com

# Cluster gossip
ClusterSize: 3
DiscoverViaDns: false
GossipSeed: node2.project.example.com:2113,node3.project.example.com:2113

Client connection string

Use the external hostname in the client's connection string rather than the gossip seed from the node configuration.

gRPC

esdb://public1.project.example.com:2113,public2.project.example.com:2113,public3.project.example.com:2113

TCP

GossipSeeds=public1.project.example.com:2113,public2.project.example.com:2113,public3.project.example.com:2113;

Note: This is assuming that your external certificate matches your wildcard certificate (e.g. node1.project.example.com for internal, public1.project.example.com for external). If it does not, then add the external hostname to the certificate's SAN.

Renewing a node certificate

In both cases of certificates issued by a public and private CA the procedure is the same for each node:

1. Issue the new certificates
2. Change the configuration file
3. Reload the node's configuration

Do these steps on one node at a time, ensuring that the cluster is stable before moving on to the next node.

Issue the new certificates

Issue the new node certificate in the same manner as before.

Copy the public key to the same locations as the CertificateFile configuration setting e.g.: /etc/eventstore/certs/node.new.crt

Copy the private key to the same location as the CertificatePrivateKeyFile configuration setting e.g.: /etc/eventstore/certs/node.new.key

Change the configuration file

Update the certificate configuration file to point to the new node certificate

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.new.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.new.key
Force a reload of the certificate configuration

Reload the node's configuration

You can reload the certificate configuration without restarting the node by either issuing an HTTP request to the admin endpoint, or by sending a SIGHUP on linux.

curl -k -X POST -u [user]:[password] --basic https://node1.project.examples.com:2113/admin/reloadconfig -d '' -v

Note: the -k forces curl to skip verification of the certificate revocation

Once the configuration has been reloaded, the server will contain log entries like:

[14476,74,19:15:23.591,INF] Reloading the node's configuration since a request has been received on /admin/reloadconfig.
[14476,23,19:15:23.593,INF] Loading the node's certificate(s) from file: "/etc/eventstore/certs/node.new.crt"
[14476,23,19:15:23.628,INF] Loading the node's certificate. Subject: "CN=eventstoredb-node", Previous thumbprint: "DA090ED6844C15F11031A1CE8A3841ECAEA5AA6C", New thumbprint: "DA090ED6844C15F11031A1CE8A3841ECAEA5AA6C"
[14476,23,19:15:23.628,INF] Loading trusted root certificates.
[14476,23,19:15:23.631,INF] Loading trusted root certificate file: "/etc/eventstore/certs/ca/ca.crt"
[14476,23,19:15:23.631,INF] Loading trusted root certificate. Subject: "CN=EventStoreDB CA 8a9b87f0cedb5230569473bc8bcca45c, O=Event Store Ltd, C=UK", Thumbprint: "24BEB8402D30DD7B62FC3DAF4F877D8911FD8CED"
[14476,23,19:15:23.635,INF] Certificate chain verification successful.
[14476,23,19:15:23.635,INF] All certificates successfully loaded.
[14476,23,19:15:23.635,INF] The node's configuration was successfully reloaded

Note: If the certificate fails validation when it is loaded up, the configuration changes will not take effect. You can then correct the certificate and try again.

Replacing a node

Replacing a node with the same hostname

Typically, you would replace a node by doing the following:

• Issue a new certificate for the new node (node4)
• On the new virtual machine:
• Install EventStoreDB. The process may not be started yet.
• IP address of 127.0.20.4
• Create a configuration file as described above, using the hostnames of the remaining nodes as the gossip seed. e.g node1.project.example.com:2113,node2.project.example.com:2113
• Restore a backup of the data and index files
• Kill the node to be replaced (node3)
• At this point, the cluster will be in a degraded mode, but still functional
• The killed node should be reported as “Dead” in the EventStoreDB UI, and from the gossip endpoint
• Change the old node’s DNS entry to point to the new IP address e.g. node3.project.example.com. 300 IN A 127.0.20.4
• If you are using AdvertiseHostToClientAs, update the public DNS entry to point to the new IP address. e.g. public3.project.example.com. 300 IN A 127.0.20.4
• Start the new node (node4)
• If you are using Cluster DNS for your clients, update the cluster DNS records to swap out the IP address of the old for the IP address of the new node. In our example, the resulting Cluster DNS would be:
• cluster.project.example.com. 300 IN A 127.0.20.1
• cluster.project.example.com. 300 IN A 127.0.20.2
• cluster.project.example.com. 300 IN A 127.0.20.4

Using this method, you do not need to update any configuration for the other nodes or for the clients connected to the cluster.

Replacing a node with a different hostname

If you want to replace a node with one with a different hostname, then you need to follow these steps:

• Prepare a new DNS entry for the new node. e.g: node4.project.example.com. 300 IN A 127.0.20.4
• If you are using AdvertiseHostToClientAs, then you will also need to prepare a new DNS entry for the public address. e.g. public4.project.example.com. 300 IN A 127.0.20.4
• Issue a new certificate for the new node
• On the new virtual machine:
• Install EventStoreDB. The process may not be started yet.
• Create a configuration file as described above, using the hostnames of the remaining nodes as the gossip seed. e.g node1.project.example.com:2113,node2.project.example.com:2113
• Restore a backup of the data and index files
• Kill the node to be replaced (node3)
• At this point, the cluster will be in a degraded mode, but still functional
• The killed node should be reported as “Dead” in the EventStoreDB UI, and from the gossip endpoint
• Start the EvenStoreDB process on the new node (node4)
• Update the configuration on the other two nodes so that the gossip seed reflects the new node's hostname. e.g. on node 1 the gossip seed would be node2.project.example.com:2113,node4.project.example.com:2113.
• You do not need to restart the other nodes in the cluster. This configuration change will take effect if the nodes are shut down for any reason.
• If you are not using Cluster DNS for your clients, update the connection string for the client applications. For example:
• Using AdvertiseHostToClientAs: esdb://public1.project.example.com:2113,public2.project.example.com:2113,public3.project.example.com:2113
• Otherwise: esdb://node1.project.example.com:2113,node2.project.example.com:2113,node4.project.example.com:2113
• If you are using Cluster DNS for your clients, update the cluster DNS records to swap out the IP address of the old for the IP address of the new node. In our example, the resulting Cluster DNS would be:
• cluster.project.example.com. 300 IN A 127.0.20.1
• cluster.project.example.com. 300 IN A 127.0.20.2
• cluster.project.example.com. 300 IN A 127.0.20.4

After replacing the node, the old node will still show up as DEAD in the gossip endpoint and the UI for some time, depending on the DeadMemberRemovalPeriodSec configuration setting. The default setting is 30 minutes.