Kurrent Cloud: Introducing Shared Infrastructure
Recently, we launched our new Shared Infrastructure feature, marking an important milestone in the evolution of Kurrent.
I'm excited to share details about our new platform!
The catalyst for change...
In the fall of 2023, I joined Kurrent to play a key role in shaping the company's long-term technical vision. At the time, our legacy cloud infrastructure was primarily focused on wrapping and exposing database deployments via virtual machines across AWS, GCP, and Azure. Essentially, the database was treated as a black box and the control plane provided the gearing to orchestrate the provisioning of cloud infrastructure.
From my experience working at innovative big-data companies I saw Kurrent's exciting potential to revolutionize the database and streaming space. For that to happen though, we first needed a flexible platform with oodles of scale potential to build upon.
The word platform ignited thoughts of:
- Resilient Scalability (both horizontally and vertically)
- Symbiosis with hosted deployments
- Security as a first-class concern
- Simplicity in deploying, testing and updating applications
- Ease of observing and monitoring constituent parts
- Multiple deployment modalities
The topic of deployment modalities is important, so what do I mean by that term?
It is segmented into three parts:
- Fully Hosted: Kurrent manages cloud deployments within its network for customers. This comes in two flavors, Shared and Dedicated.
- Shared (what this blog is about), is a Kubernetes cluster owned and managed by Kurrent.
- Dedicated is for customers needing their own Kubernetes cluster but running inside of Kurrent’s network.
- Bring Your Own Cloud (BYOC): Customers can use their own cloud provider account to host Kurrent data plane components on a dedicated Kubernetes cluster. This option is perfect for folks needing data sovereignty but still benefit from the control plane hosted by Kurrent.
- Hybrid: Kurrent manages the control plane in the cloud, and customers can use a Kubernetes cluster hosted on-premise.
Comparing all the items defined above with our strategic vision revealed that moving to a Cloud-native architecture was crucial for our future advancement.
Evolution
Technology selection can be difficult, but in the Cloud-native space, there is one clear winner. Kubernetes is the modern Enterprise standard for deploying containerized applications at scale. It's supported by all the major cloud providers and globally adopted. From Kurrent's perspective it offers compelling benefits, such as:
- Abstraction over infrastructure: We let Cloud providers focus on native compute/storage provisioning so that we don't have to. This allows us to leverage a standardized Kubernetes resource API.
- SaaS cost efficiency: The ability to fully utilize resources is crucial for us as a company and our customers.
- Ease of consumption: Production-grade clusters can be run in the Cloud or on-premise.
- Elasticity: Kubernetes was purpose built with scalability and complex orchestration in mind.
- Mass adoption: Kubernetes has been battle tested in large production environments over a long period of time.
- Wide community support: Kubernetes is a highly active project and has support from many vendors focused on:
- Development/testing tools e.g. Kind, K3S, IDE integrations, etc.
- Storage drivers
- Networking drivers
- Monitoring
- Large scale data processing, such as big-data frameworks like Spark
- ...and more!
Architecture
The platform architecture is split in two fundamental parts, the control plane and data plane.
Each plane is really a Kubernetes cluster that hosts Kurrent containers to form a distributed architecture. There can be many data plane clusters where each is composed of logical sites that are uniquely identifiable. A site is synonymous with a Kubernetes namespace and a customer will have one or more of these. You can think of a site as a private and isolated space for containers to run.
The diagram illustrates this concept at a high level:
Controllers form the backbone of the architecture and work together behind the scenes to perform jobs and handle commands, namely:
|
Name |
Plane |
Description |
|
Root Controller |
Control Plane |
The main platform brain used to keep track of clusters and sites (and yes, it is event sourced!) |
|
Cluster Controller |
Data Plane |
Used for managing a site's lifecycle |
|
Site Controller |
Data Plane |
Used for managing deployments e.g. KurrentDB |
Communication
Communication between controllers is secured with TLS. Authorization is managed via a JWT that is provided by our internal Identity Provider after successful service account authentication. All ingress traffic within the control plane and data plane is managed by multiplexed proxies.
IMPORTANT: By design, controller communication will only ever reach out to the root controller (call home strategy). As we head towards offering Hybrid deployments, customers will not be required to change their inbound traffic rules!
Both cluster and site controllers routinely heartbeat back to the root controller and check for the next command to execute as illustrated below:
What is a command?
Commands serve as a mechanism to perform some type of remote action. With the release of Shared Infrastructure, the list below highlights some of the commands supported today:
- Deployment of KurrentDB in a site (with a given version, configuration, etc)
- Removal of KurrentDB from a site
- IP address updates for KurrentDB deployments
- Toggling mTLS on KurrentDB deployments
High Level View
Today, the platform architecture largely reflects the figure above (more Cloud providers will be added very soon). Our Shared Infrastructure model simply means that Kurrent own and manage the Kubernetes cluster that is shared between customers.
Some key points to highlight here include:
- Data plane components only ever call home, no inbound traffic from the root controller
- Customers leverage isolated namespaces (sites)
- The root controller and supporting components leverage Kubernetes native horizontal-pod auto-scaling capabilities
- Cluster auto-scaling is used to manage the physical shape of the Kubernetes cluster
- Ingress traffic is multiplexed through proxy controllers
- Prometheus is used to collect and dispatch metrics/logs to Grafana
Why Shared Infrastructure?
Shared Infrastructures offers the perfect entry point for getting started with Kurrent. This latest offering brings new ways to deploy KurrentDB using public networking, and for advanced users, the ability to leverage mutual TLS for highly secure environments.
We offer three deployment tiers that are focused on the journey from development to production:
S0: Sandbox Environment
If you're looking to get started with KurrentDB, this offers all the bells and whistles that KurrentDB provides running on a single node (projections, subscriptions and beyond).
S1: Development Environment
A seasoned developer with KurrentDB can use this tier and begin scaling up to use multiple-nodes.
S2: Small Production Environment
This tier is perfect for deploying and running small production workloads with minimal resource requirements.
What's next?
The roadmap for 2025 is packed with goodies in the near term, such as:
- Google Cloud and Azure supported as providers on Shared Infrastructure and beyond
- Backup/Restore for KurrentDB clusters (Shared Infrastructure and beyond)
- Dedicated Kubernetes clusters for Fully Hosted customers
- BYOC – Bring your own cloud
Stayed tuned for more things to come! Head on over to Kurrent Cloud and get started today.