Kubernetes multi-tenancy involves creating per-tenant instances of an application ensuring isolation between the instances. It starts with a Helm chart for the application deployment. Platform engineering challenge is to convert this deployment workflow into a service that can be used repeatedly for every tenant with required controls and tracking. Platform-as-Code is a framework to design such multi-tenant platform services from Helm charts.
Stack level multi-tenancy
In this scenario, platform teams prepare the cluster for running multiple instances of an application stack. It requires platform service/s to simplify repeated deployment of the stack with required isolation, CPU/memory resource allocation and monitoring. Platform-as-Code can be used to create Kubernetes APIs to deliver such platform services.
Team level multi-tenancy
In this scenario, platform teams prepare the cluster to be used by various internal product teams. This involves creating different users and roles with appropriate privileges in addition to offering multi-tenant application services for these teams. Platform-as-Code can be used to deliver services in such multi-user environments as well.
CRD for CRDs
Design your platform services from Helm charts
Resource relationship graphs
Visualize your platform workflows
CRD for CRDs
KubePlus offers a CRD named ResourceComposition to
- Compose new CRDs (Custom Resource Definitions) to publish platform services from Helm charts
- Define policies (e.g. Node selection, CPU/Memory limits, etc.) for managing resources of the platform services
- Get aggregated CPU/Memory/Storage Prometheus metrics for the platform services
Resource relationship graphs
Platform workflows are realized by establishing relationships between available Kubernetes Resources / APIs (built-in or Custom). These relationships are primarily of four types - (1) Owner references, (2) Labels, (3) Annotations, (4) Spec Properties. KubePlus is able to runtime construct Kubernetes Resource relationship graphs. This enables KubePlus to build resource topologies and offer fine grained visibility and control over the platform service.
Building a sample platform service
Here we demonstrate how a platform team can build a MySQL service for their product team/s to consume. The cluster has base Kubernetes and MySQL Operator installed.
The platform workflow requirements are:
- Create a PersistentVolume of required type for MySQL instance.
- Create Secret objects for MySQL instance and AWS backup.
- Create MySQL instance with backup target as AWS S3 bucket.
- Setup a policy in such a way that Pods created under this service will have specified Resource Request and Limits.
- Get aggregated CPU/Memory metrics for the overall workflow.
Here is a new platform service named MysqlService as Kubernetes API.
A new CRD named MysqlService has been created here using ResourceComposition. You feed a platform workflow Helm chart that created required underlying resources, and additionally provide policy and monitoring inputs for the workflow. The Spec Properties of MysqlService come from values.yaml of the Helm chart.
Product teams can use this service to get MySQL database for their application and all the required setups will be performed transparently by this service.
Platform Service in use
Resource relationship graph of MysqlService instance
Here is a visual representation of the complete resource relationship graph of the MysqlService instance. This can be discovered using KubePlus kubectl plugin - 'kubectl connections MysqlService mysql1'. You can see that the platform workflow specified in the Helm chart gets deployed when the instance of the MysqlService is created along with the specified policies and monitoring inputs.
CPU utilization in Prometheus for MysqlService instance
KubePlus provides aggregated CPU/Memory/Storage metrics in Prometheus format for the service that can be discovered in your monitoring infrastructure.