Kubernetes extensions (commonly referred as Operators) extend Kubernetes Resource set (APIs) and enable adding application specific workflow automation in Kubernetes native manner e.g. Operators for softwares like databases, key-value stores, API gateways etc. Enterprise DevOps teams use Kubernetes Operators and create their Kubernetes native stacks.
Platform workflows are realized in Kubernetes YAMLs by establishing connections between Kubernetes Resources (APIs). These connections can be based on various relationships such as labels, annotations, ownership etc.
How to build application workflows using available Kubernetes Resources (APIs)?
If a node fails, what all services will get affected?
How much CPU/Memory/Storage is consumed by the entire application?
Enterprises are building their Kubernetes native stacks using a variety of Kubernetes API extensions but then struggle to manage platform workflows created using the available APIs. At CloudARK we are pioneering Platform-as-Code technology that enables users to visualize and troubleshoot these platform workflows. It involves tagging Kubernetes API extension packages with a unique method (Platform-as-Code annotations on CRDs) that unlocks our open source KubePlus tooling to work for that specific stack. This approach is generic without any tie-in with API extensions, Kubernetes distributions or cloud providers. The approach is also agentless and does not require any code to be running on the Kubernetes cluster.
Mechanisms used to establish Kubernetes Resource relationships include - labels, annotations, spec properties and owner references. When working with Custom Resources (APIs), it is important that assumptions made by its developer around what relationships can be established with a Custom Resource and what actions will be performed as a result of them are clearly articulated. KubePlus provides a set of annotations on Custom Resource Definitions (CRDs) to encode such assumptions.
KubePlus leverages knowledge of relationships between Kubernetes built-in resources and combines that with the CRD annotations. It dynamically constructs Kubernetes resource relationship graphs for various application workflows running on a cluster. KubePlus offers number of kubectl plugins that internally leverage these graphs and enable teams to visualize and troubleshoot platform workflows.
Here is a simple example of how a CRD annotation on ClusterIssuer Custom Resource enables KubePlus to show various resource relationships in its workflow.
ClusterIssuer Custom Resource works based on annotation relationship on Ingress resources. So we have annotated this ClusterIssuer CRD with the following Platform-as-Code annotation:
resource/annotation-relationship: on:Ingress, key:cert-manager.io/cluster-issuer, value:INSTANCE.metadata.name
This enables KubePlus to show all resource relationship graphs involving ClusterIssuer Custom Resource as shown below:
$ kubectl connections cr ClusterIssuer wordpress-stack namespace1
::Final connections graph::
------ Branch 1 ------
Level:1 Ingress/wordpress-ingress [related to ClusterIssuer/wordpress-stack by:annotation]
Level:2 Service/wordpress [related to Ingress/wordpress-ingress by:specproperty]
Level:3 Pod/wordpress-pod [related to Service/wordpress by:label]
Any ‘as-Code’ system is designed to offer a common declarative language to provision and manage a technology stack leveraging underlying APIs / resources. In the Kubernetes world, Kubernetes YAML itself supports the use of Custom API Resources and hence a language for platform automation exists. The challenge is in discovery and consumption of variety of Custom API resources, in other words management of this platform automation. At CloudARK, we are pioneering Platform-as-Code approach that enables DevOps teams to visualize and troubleshoot Kubernetes platform workflows. It involves curating Custom Resource Definitions (CRDs) in your stack with a few annotations. It unlocks our KubePlus tooling to work specifically for that specific stack. The tooling significantly simplifies the process of managing the Kubernetes platform workflows.
In order to build Kubernetes application workflows using Operators and related Custom Resources, it is important for Cluster administrators to evaluate community Operators against a standard set of requirements. We have developed Operator Maturity Model for this focusing on Operator usage in multi-Operator environments.