OpenShift Operators vs. Helm Charts: When to Use Each

The approach to cloud natives in Red Hat OpenShift depends on the implementation of complex technology stacks to package, deploy, automate, and maintain highly complex distributed systems. In order to obtain knowledge required to perform these processes, it is necessary to go through the openshift training courses provided by OnlineITGuru. Among many different combinations of basic technologies implemented by platform engineers, two approaches for deploying software can be noted: Helm charts and OpenShift Operators.

Although it may seem to many newcomers that both technologies solve one and the same problem and can be treated as some kind of rivals, everyone who knows something about Kubernetes and OpenShift should be aware that these are totally different kinds of technologies that work on totally different levels. Namely, Helm is a technology of packaging, automation, and templating Kubernetes resources, while OpenShift Operators are software extensions that embed human operational knowledge in the form of control loops.

The choice between them or their combination requires deep understanding of their architectures, life cycles, maintenance costs, and security features.

Decomposition of Helm: The Kubernetes Package Manager

During many years, Helm has been widely used as the official package manager in the whole Kubernetes ecosystem. With such a design, which is inspired by operating system package managers, like Debian package manager or Homebrew on macOS, Helm provides a solution for packaging, configuring, deploying, and versioning the bundle of manifest resources.

Architecture and Functionality

As for the definition, the Helm package can be considered as a number of files with a specific structure packaged in the package called chart. Such a chart contains metadata about its name and version, configuration value schema and its defaults, and manifest templates implemented using Go templates.

During installation and upgrade of a chart in the OpenShift cluster, Helm retrieves configuration values, merges them with custom configuration values, evaluates the templating engine and generates a bundle of fully rendered Kubernetes manifests that are sent to the OpenShift API server through HTTP requests.

Thus, Helm will verify manifests, authenticate the request, use role-based security policies, and ensure the persistence of the desired state defined in the key-value storage of the cluster. Notably, once the manifests are transferred and the release metadata (commonly stored as secrets within the target namespace) is persisted, the process of working with Helm is completed. Helm will not continuously verify whether the resources' states released match each other unless explicitly triggered by a human user or CI.

Operational Features of Helm

There are several features provided by Helm for better operational experience, which are particularly relevant to stateless services, packaged applications, and common multi-environment deployments:

Simplicity and Usability: Building and editing Helm charts involves template creation of manifests mostly. The platform team that is well versed in Kubernetes constructs such as Deployments, Services, Ingresses, and ConfigMaps can create Helm charts without learning object-oriented programming and deep knowledge of the control plane.
Variable Substitution: Helm excellently executes variable substitution among different environments. All you need to do is pass values files for the same base chart in the dev, test, staging, and prod clusters.
Versioning and Rollbacks: According to Helm, all deployments are different versions of releases and have their version number. In case there are any regressions in the configuration, Helm can roll back all release changes to the previous version. Helm will compare the previous version of the release with the new one and make the necessary patches for the cluster API.
Adoption within the Cloud-Native Ecosystem and Reusability: Cloud-native ecosystem keeps plenty of publicly accessible Helm charts of popular open-source software solutions from message brokers to observability stacks to web servers. Using a Helm chart will help you to reuse manifests and not create them anew when deploying a third-party solution.

Weaknesses of Helm

Helm is only a package and template manager and lacks additional functionality. Helm does not care about what is going to happen to the application after the deployment of the manifest:

Lack of the Continuous Reconciler: Helm renders manifests, deploys them and stops working. If someone modifies a Deployment or removes a Service which was created by Helm, Helm will not detect it until the next deployment.
Inability to Accomplish Stateful Day Two Operations: High-level, stateful systems such as relational databases, consensus-based protocols, or enterprise-grade message queues are not usually implemented in a sequential manner. There are too many intricate processes in play – rolling upgrades across a multi-node cluster, database schema migration, quorum aware failover, backup and restore – to represent them using a templated format.
Client Initiated Actions: The Helm method presupposes the existence of an external party, which starts the procedure. No matter if it is a laptop of a developer or a pipeline runner – it is the intellect behind the operation, which operates from outside of the cluster control plane.

Breaking Down the Operators: Codifying SRE Practice

The idea of an Operator is definitely a great improvement over package management. It was introduced to compensate for a difference between human operators' skills and mechanisms of the cluster, making Kubernetes control plane itself the one to manage the applications.

Operators form the basis of Red Hat OpenShift operating system. OpenShift runs all of its components, including networking stack, DNS, storage drivers, authentication and monitoring mechanisms, and control plane itself using Operators managed by Cluster Version Operator.

Core Architecture & Mechanics

Operator consists of two main components: Custom Resource Definitions and custom controller, which runs in reconciliation loop:

Custom Resource Definitions: Custom Resources are extensions to OpenShift API schema. By using Custom Resource it’s possible to create a domain object. Rather than orchestrating a number of low-level objects the developer uses a high-level intent expressed in a domain object.
Controller and the Reconciliation Loop: Controller is a proactive daemon which operates inside the cluster. It checks the current state of the managed application and compares current state with the desired state which is specified in Custom Resource. Then the controller executes the necessary actions to synchronize the current reality with the desired state.

Such control loop operates continuously. In case if a node fails, network split fixes itself, persistent volume gets corrupt or the unauthorized user modifies a sub-resource the operator will notice the difference and execute the actions to get the desired state restored.

Capability Model for Operators

In order to determine the complexity of an Operator industry uses maturity operational model. The model consists of five levels:

Level 1 (Basic Install): Automation of installation and basic configuration of the application workload.
Level 2 (Seamless Upgrades): Automation of patching and upgrading, format conversions and rolling upgrades.
Level 3 (Full Lifecycle): Complete lifecycle automation including automated backup, failure recovery, scalable storage and automated high availability failovers.
Level 4 (Deep Insights): In built observability, automated metrics collection, alerts validation for the application and automation of performance self healing.
Level 5 (Auto-Pilot): Automation of operation which includes horizontal/vertical scaling with custom business metrics, proactive anomaly detection and resources management.

Operator Framework and Lifecycle Management on OpenShift

The OpenShift platform is an intelligent one, and it provides capabilities for deploying and managing the Operators using Operator Lifecycle Manager. The sub-system manages the distribution, dependency management, upgrades, security isolation, and multi-tenancy of all operators on the cluster.

Operators have been incorporated into the OpenShift web console through OperatorHub, which enables cluster operators to find, install, and configure enterprise operators with merely a few clicks. The Operator Lifecycle Manager monitors the installed operators for catalog upgrades, applying security patches or feature upgrades as per the approval policies set up by the cluster administrator.

Advantages of Operationalization with Operators

Real Autonomic Operations: Operators incorporate human operational knowledge into code. Operational domain knowledge that defines what must happen in certain scenarios, for example, selecting a new leader for a database in case of node eviction, balancing of distributed storage shards, and orchestrating a zero-downtime certificate rotation, will run autonomously without paging an on-call operator.
Continual Drift Mitigation: As long as the controller is doing the reconciliation, any drift from the desired authorized configuration either accidentally or intentionally will be identified and reversed.
Integrations within the Platform: Operators within OpenShift are fully integrated into the platform capabilities like routes, service mesh capabilities, logging, storage classes, and image registry.
Standardized Consumption of Platform APIs: Using complex applications modeled through custom resources, the platform team makes it easy for the platform users to deploy applications without accessing dangerous low-level manifest files.

Costs and Trade-offs for Operators

Although the Operator pattern has many benefits, here are some of the costs and trade-offs associated with this pattern:

Costs in Software Engineering: Development of the operator is a costly process from the point of view of software engineering since the developer is supposed to code controllers in programming languages such as Go, do version API management, deal with edge case failures and understand distributed synchronization primitives.
Requirement for Increased Privileges at the Cluster Level: Given that the controllers manage various resources in their life cycle, the controllers need increased privileges at the cluster level.
Maintenance Efforts Required: An Operator is a software which needs maintenance by way of patching, vulnerability scanning, unit testing and new API deprecation in Kubernetes.

Detailed Comparative Analysis

In order to understand what paradigm would be more appropriate for particular loads, organizations have to conduct comparative analysis of both of the approaches taking into consideration relevant operational parameters.

Lifecycle Management: Day One and Day Two

The split of Day One (set up process) from Day Two (management, patching, scaling, recovery) appears to become the key difference between Helm and Operators.

Helm works perfectly in Day One case since it always takes an application from nothing to running state by using the identical set of manifests. In the case of the shift of application to the Day Two stage, Helm is no longer useful since it is unable to independently monitor the lagging of database replication, drain connections before a major update, and conduct a staged schema migration after pod rotation.

At the same time, Operators were specifically designed for the Day Two case. While they still work perfectly fine in Day One case, it is in the Day Two case where their potential shows itself to its full extent.

Security, Permissions, and Governance

To begin with, it should be noted that there is a huge difference between the architecture of these tools, taking into account their multi-tenant and regulated nature of OpenShift.

When executing actions with Helm, everything takes place within the security boundaries of the user. If this action is executed by a junior developer, OpenShift checks permissions of this developer. When the permissions of this particular developer do not permit creating a cluster-level resource, everything will be secure and this action will not be executed. This way of execution of actions on client-side perfectly suits the idea of access control and least privilege principle.

When executing actions, Operators work on server-side using the wide permissions of persistent Service Accounts in certain namespaces or even the entire cluster. However, when the Operator controller is compromised, an attack can be based on these wide permissions. Although OpenShift defends against such attacks using security context limitations and OLM policies, security experts still have to check the permissions of the Operators before installing them.

Configuration Drift and Reconciliation

In traditional IT processes, configuration drift is described as the presence of a difference between the real state and the documentation of the environment.

Configuration drift with Helm is carried out through a passive approach. In case there is any issue in the production environment that makes the engineer change the port of the service or memory limit by running commands in the command line interface, it will only be detected during the next step by Helm. This may be overwritten by a continuous delivery system that leads to even greater issues.

Configuration drift is actively eliminated through the use of operators. The reconciliation loop will look for differences between the real state and the definition of the custom resource many times per minute.

Observability and Diagnostic Capability

On the one hand, Helm can provide no observability because it just checks whether all the pods have been made ready to pronounce the release to be successful. On the other hand, Helm does not know anything about the internal health state of the application which runs inside those pods.

One can provide application-specific metrics to Prometheus stack of OpenShift, the status of their custom resources, and global cluster events to be able to diagnose the health state of the application. When an Operator operates distributed cache, it may provide custom status of cache warming percentile, shard health, and synchronization latency metrics.

Strategic Decision Making Framework: When to Choose What

But this should never be looked at as a religious conflict between Helm and the Operator. The thing to remember here is that the individual must take into consideration the type of workload, the experience of the operation team, and how it affects the cost of maintenance in future years. As teams go about creating their own operators for the Day Two operations, they find it beneficial to take up an openshift certification course at OnlineITGuru.

When Helmi is Superior to Operator

There are cases when Helm beats Operator in terms of simplicity and low deployment/maintenance costs:

Stateless Web Services and Microservices: Stateless web services, microservices and client-side services which do not contain any state information inside of them do not require automation of the lifecycle management process. Existing objects like Deployments, Horizontal Pod Autoscaler, RollingUpdate should be enough from the reliability perspective. Development of an Operator for such applications will be useless.
Standard CI/CD Pipelines: Enterprise level organizations which already have advanced CI/CD Pipeline will find Helm quite useful as a package manager to manage the tested, version-controlled, parametrized and release through environments changes of the application using automated pipeline.
Prototyping, Testing and Temporary Environments: If it is necessary to deploy applications extremely fast, Helm charts can be useful. It could be used to create temporary environment which can be used to test branches or prototype without development of any custom APIs and controllers.
Third Party Tools “Out of the Box” with Simple Clustering: Some third-party tools which require minimum configuration and some ingress paths and volumes will be installed using Helm Charts.

Use Cases of OpenShift Operator

The operator should be used in scenarios when human interaction is required to ensure the health, consistency, and performance of the application:

Stateful, clustered and distributed applications: Databases, messaging queues, storage engines, search clusters, and any other applications require proper execution of procedures such as installation, scaling, failover, and maintenance. Operator understands how to execute the correct sequence of the procedure protocols that guarantee data consistency and cluster quorum.
Critical enterprise systems: When critical enterprise systems exist, in which no downtime is tolerable during minor and patch releases, Operator can automate multiple steps of rolling updates, data migration, and sanity checks before updates that generally require a whole runbook and human interaction.
Engineering platforms and self-service portals: If central platform teams provide database-as-a-service or messaging-as-a-service to developers, then the Operator is the proper abstraction level to choose. Developer gets the enterprise database with the help of just one configuration file for custom resource.
Reconciliation Process for Aggressive Prevention of Drift: In situations where there are highly regulated industries and any drift in the environment runtime configuration may lead to non-compliance and/or some security concerns, aggressive reconciliation process by Operator would resolve the issue.

Coexistence Approach: Combining Both Approaches

This misconception stems from an erroneous assumption that organizations have to opt either for Helm charts or Operators due to different means of microservice management. What needs to be mentioned here is that within a complex architecture on Red Hat OpenShift, the two approaches can be applied together.

Within the combined approach, the difficult task of installation and management of Operator will be performed by cluster administrator through Operator Lifecycle Manager. Operator will be enriched with additional resources.

As a result, both applications and development teams will create Helm charts which will include not only their microservices but also additional resources. For instance, Helm chart created by applications developers will define stateless deployment of application, its Service, its OpenShift Route and custom resource which would instruct PostgreSQL Operator to manage a particular database for a particular application.

It is here that Helm comes in to package, promote, and organize the application layer that the team has been working on, and it is here that the Operator comes in to address the complexities of the database layer underneath. In this scenario, each is utilized based on its strengths; while Helm does the packaging part which is simple and lightweight, the Operator does the lifecycle engineering for the autonomous runtime.

Conclusion

And there we have the answer to the question of how they differ from each other; one is responsible for packaging software while the other is responsible for operating the software.

Helm makes for an excellent choice as far as packaging your software is concerned because it takes care of all the necessary processes in order to translate templates to parameterized delivery into the environment in the most efficient manner possible.

OpenShift Operators, on the other hand, represent living entities that operate together with the applications inside the cluster. They transform administrative runbooks into infinity loops of code. They work extremely well in case of state, clustering, proper ordering, and maintenance, which requires continuous management. They give the answer to the critical Day Two question: How to keep the complicated and dynamic software alive, update, fix, and manage for a few years?

The engineers will be able to dissolve false dilemmas, taking into consideration the complexity of the applications, their states, and capabilities of the organization. Combining Helm for good configuration and operators for runtime operation, one will create the OpenShift platform that works well for developers and for operations.