Enterprise Disaster Recovery and Multi-Cluster Resilience on OpenShift

The Importance of Architecture for Resilience

Cloud-native modern design has shifted the paradigm in disaster recovery dramatically. Disaster recovery in typical infrastructure was mainly about the creation of warm or cold infrastructure, and the practice of exporting the snapshots at hypervisor levels. By deploying the distributed systems on the Red Hat OpenShift platform, the complexity of business continuity and opportunities associated with it were heightened.

OpenShift cluster can be called an autonomous system performing its duty thanks to the work of an orchestration fabric consisting of various components such as API definitions, runtime configurations, software-defined networks, etc. Mastering these complex primitives is the major objective of a standard red hat openshift course because administrators should know how various components function before considering the issues of business continuity. The fact of separation of physical infrastructure from workloads enables one to discuss disaster recovery on two levels: on the one hand control plane metadata is responsible for the operation of services and the second one marks the storage of data.

In order to create an enterprise-level disaster recovery approach, it is necessary to define clearly how services should be expected to operate. There are two performance indicators which determine all important architectural decisions for a given solution: recovery point objective (RPO) and recovery time objective (RTO). RPO stands for the maximum acceptable level of data loss, and it is expressed in the timeframe during which data should be fresh after recovery has occurred. At the same time, RTO defines the maximum time period allowed between the time of outage announcement and the complete provision of the service.

It should be noted that achieving the maximum performance would require difficult compromises in terms of capital expenditures, network capacity, operational costs and complexity of the given project. The multi-cluster approach that guarantees zero RPO across hundreds of miles will require synchronous storage mechanisms, practically zero latency and sophisticated traffic engineering. On the other hand, if RPO can be measured not in minutes, but in hours, the architecture can use point-in-time snapshots of the state that are saved asynchronously into cold object storage.

Analyzing Platform State and Workload Structures

A resilient platform requires analysis of the manifestation of state in an OpenShift environment. Applications running in OpenShift are not often standalone processes. Rather, any production service comprises a constellation of the Kubernetes components: Deployments, StatefulSets, Custom Resource Definitions, ConfigMaps, Secrets, Role-Based Access Controls, and network policies, among others. Collectively, these manifests form the definition of the application in question.

At the same time, the control plane is managing another layer of cluster visualization including oauth identity, machine configs, ingress controller definitions, and operator subscription states. This is all the control plane metadata is stored in the internal etcd key-value storage. Hence, when it comes to developing recovery architectures, people often confuse the recovery of etcd with the recovery of an application. An etcd snapshot exists to restore a particular state of the cluster control plane, which might not be applicable for cross-cluster migration, granular failovers, and disaster recovery of persistent volumes.

The persistence layer creates a new problem to solve. Stateless microservices depend on external backing technologies such as cloud databases, object stores, or enterprise database systems. Stateful microservices, on the other hand, will use Block Storage via Persistent Volume Claims (PVCs) and through Container Storage Interface (CSI) drivers.

In the case of an outage on a physical datacenter, cloud availability zone or a regional edge site, restoring container runtime without a perfect recreation of the attached persistent block device will lead to a non-operational service. Thus, the aim of this architecture is to create an operational pipeline to capture the runtime metadata, create a storage parity across the sites and control the routes guiding client requests to the successfully functioning workloads without human intervention.

Stateful Backups and Metadata Protection with aid of OADP

The basis for point-in-time recovery and cross-cluster portability in OpenShift is the OpenShift API for Data Protection (OADP). Built on top of the upstream project Velero, OADP offers an operator-centric framework tailored to the OpenShift ecosystems. It interacts natively with the OpenShift custom resources, security context constraints, and its underlying CSI storage plugins to carry out coherent application-level backups.

The work of OADP takes place inside the cluster, thanks to the unique deployment controller that registers Custom Resource Definitions, with the Backup, Restore, BackupStorageLocation, and VolumeSnapshotLocation as the main resources. Unlike other storage snapshots, OADP inspects an API server of the cluster when a backup takes place. It processes the dependency tree of Kubernetes resources and generates a list of the namespaces, labels, and other objects in a declarative form.

In order to target persistent information related to these items, extensive integration with storage hardware and files is needed. OADP is capable of achieving that thanks to two different methods: Container Storage Interface VolumeSnapshot API and direct copying tools (like Kopia and Restic).

In the case of CSI plug-ins being utilized, OADP guides the storage provider so that a block-level snapshot is taken. The snapshot keeps the storage array at the moment of taking it. When dealing with storage systems that have poor CSI snapshotting capability or hybrid situations where persistent volumes are moved across different backend storage types (with different approaches sometimes needed), OADP would take file system backups, with the agent being the privileged DaemonSet that will go through the hierarchy of the mounted volumes and copy the blocks that were changed to a specified backup location.

In order to record the lasting state in situations where data is constantly changing, the applications must undergo a phase of quiescence. If the enterprise database is constantly writing transaction logs in memory buffers and file caches during the time when a snapshot is created, it is possible to obtain torn write or inconsistent snapshot during recovery. As a result, the database must perform extensive journaling recovery measures or fail to start up at all, as a result of possible structural corruption of the database.

OADP solves this problem with the help of pre-backup and post-backup lifecycle hooks. These hooks allow commands to be executed in application pods before and after the snapshot is being taken. In the majority of cases the pre-backup hook executes flush and lock command, ending the write process and forcing the dirty pages to be sent to the disk. After the initialization of the CSI snapshot, which requires very little time, the post-backup hook proceeds with unlocking the database and resuming the write process while the data replication takes place.

It is important to understand environmental parity when performing OADP state restoration across clusters. For instance, it can lead to unforeseen issues if a backup from one OpenShift cluster is restored into another OpenShift cluster without considering the differences in storage classes, service account authorizations, and network configurations. Excellent OADP operator hooks and recovery procedures are a significant step in hands-on openshift certification training and prepared engineers for real-world recovery situations.

OADP allows the use of declarative resource modifiers, which let admins use different storage classes during the restoration process. For instance, a persistent volume backed by high-performance local NVMe in an on-premises data center can be restored onto a similar cloud SSD class storage without developers needing to modify the manifest files.

OADP is a source of enterprise insurance against disasters that are not of catastrophic nature: it eliminates incidents of accidental deletion by a developer, cases of delinquencies by malware that affected local storage pools, and it even allows regional cluster catastrophes where RPO of one hour and RTO between twenty and forty minutes is acceptable for business.

The use of multi-cluster storage replication fabrics

Where Service Level Agreements require low RTOs and RPOs, relying on traditional metadata backup and object storage restoration is not a sound approach. The process of restoring numerous terabytes of data via traditional networking options simply cannot provide the needed speed of restoration. In this case, engineering teams should consider multi-cluster storage replication fabrics capable of keeping the data layer updated across the network of OpenShift clusters.

Data replication technologies can be divided into two basic types: synchronous and asynchronous. The selection made between these two options would result in an engineering trade-off between network physics and parameters of recovery.

To perform synchronous multi-cluster replication, it is necessary for the write operation to be completed on both the main site and remotely before proceeding with acknowledgement of receipt to the application. Therefore, the RPO will equal zero. The moment the data at the primary site is lost, the data at the secondary site will be preserved fully with all bits committed in the same instant. However, synchronous systems have limitations related to the speed of light in optical fibers and network switches involved in the operation.

A delay of every millisecond that occurs between the data centers directly impacts application write delay. This means that synchronous replication will only work in the case of metro distance topologies where the round-trip time is less than ten milliseconds. In such cases, Red Hat OpenShift Data Foundation (ODF) can use Ceph storage technology as part of stretched-cluster architecture or multi-cluster architecture based on external connectivity and interconnect daemons.

Asynchronous multi-cluster replication separates the verification of local write from the replication of data in remote locations. Thus, while the write is being done at the storage level and the application is being notified, background processes are used for aggregation, compression, and transferring the block-level changes to the disaster recovery storage.

The compromise indicates that RPO is less than one. In the case of an abrupt, unscheduled failure, the amount of lost data lies in the volume of all changes done in the replication cycle interval.

Current methods utilize storage operators with mirror daemons in charge of observing the process of block allocation and replaying the snapshots’ deltas in line with a timetable determined beforehand, which may range from minutes to hours. Being able to take practical lab courses with these storage and networking topologies offered by an openshift online course creates the technical basis for the successful operation of multi-cluster systems. RHACM partners with OpenShift Data Foundation using the Submariner technology to automate and manage the asynchronous relationships in storage.

Submariner plays an integral role in maintaining multi-cluster resilience and in establishing encrypted cross-cluster network tunnels on the software-defined networking level. If it had not been for this overlay-to-overlay network architecture, storage daemons within OpenShift software-defined networks would have needed to employ sophisticated ingress routing and external load balancers.

Submariner routes traffic across clusters automatically, which allows the Ceph RBD mirror daemons located in the secondary cluster to receive streaming replication directly from the primary cluster storage.

To ensure the continuity of persistent volumes in distributed environments, it is critical that the data remains synchronized. Using a passive-active replica setup, the persistent volumes in the recovery center must stay in the locked position, either in read-only or inactive condition, while the primary location is operational.

If two clusters are allowed to be active simultaneously and modify replicated volumes in the absence of a lock manager, data will be compromised immediately. To maintain the flow of operations, the failover mechanism must be orchestrated properly: persistent volumes in the active cluster must be dethroned and the locks broken.

RHACM makes this failover process automatic thanks to DR policies that link applications in the primary and backup clusters.

Global Traffic Management and Ingress Failover Strategies

A completely synchronized data layer becomes ineffective when the end-user traffic does not reach the active platform. Ingress traffic steering in distributed OpenShift clusters requires the implementation of a global routing tier that operates outside any cluster. Global traffic management utilizes two major architectures: Anycast IP routing and Global Server Load Balancing (GSLB) through dynamic Domain Name System resolution.

Anycast provides no-delay failover at the network routing level thanks to Border Gateway Protocol (BGP). In the Anycast model, the ingress controllers of both the primary cluster and the backup cluster advertise the same public IP address to upstream Tier 1 ISPs. The routers will route packets of the client based on the BGP AS path.

In the event of failure of the primary cluster, its edge routers stop advertisement of BGP routes. The upstream routers notice the route cancellation and converge, directing any subsequent packets to the backup cluster with the same public IP address being advertised.

Anycast removes the delays caused by the caching process involved in DNS-based routing. Nevertheless, Anycast calls for the direct management of enterprise IP address spaces and close coordination with telecommunications companies, which makes it difficult to implement the technology in purely public cloud environments.

Dynamic DNS-based Global Server Load Balancing (DNS-GSLB) is still the most prevalent technique for global ingress management. The GSLB platform is an authoritative DNS server in charge of e-mail domain names. Furthermore, it constantly checks the clusters’ health by transmitting queries to specific probe points that OpenShift Ingress or OpenShift Routes exposes.

As part of this process, when implementing health checks, administrators should be careful about avoiding the testing of static infrastructure endpoints such as the homepage of the platform router or an empty node. Instead, the health probes must evaluate complex synthetic endpoints that check the operational availability of ingress stacks, internal application pods, and downstream persistent storage mounts.

In case of operational failure in the primary cluster, the synthetic health probe also fails. The effect of the GSLB service is to mark the primary endpoint as down, which changes its DNS responses. The public ingress IP of the secondary OpenShift cluster is returned.

The greatest point of friction for GSLB architectures is DNS Time-To-Live (TTL) expiration. Even when a business sets a low TTL parameter on a DNS record, public recursive resolvers, enterprise firewall, and client web browsers usually ignore that value. They hold on to the old IP address for several minutes or even hours.

To solve the problem of DNS Time-To-Live (TTL) expiration, the application architecture may solve the issue of graceful retries on the client-side while platform teams need to synchronize DNS failover policies with the means of fast connection timeouts.

Not only routing traffic, but also maintaining consistent network policies is crucial element of multi-site cluster operation. Ingress controllers use TLS certificates to end a session. In case the application has failed over to a secondary cluster which does not have required TLS certificate information, clients will see warning banners regarding certificate expiration and inability to secure the session.

Using declarative GitOps processes to automate the synchronization of secrets and configurations across clusters is vital for the implementation of a global ingress failover system.

Active-Passive versus Active-Active Environments

The selection of the right topology is crucial in determining what kind of catastrophes a company can overcome. Company recovery methods lie somewhere between the extreme poles of the Active-Passive model and the truly global Active-Active topology systems.

Active-Passive Topologies

The Active-Passive architecture is the most commonly used for stateful workloads. Here, Cluster A runs the production workloads, taking 100% of the traffic and keeping read-and-write access rights to all the data saved in persistent volumes. Cluster B is created as an imitation of Cluster A which can be switched off and not have any application pods or have the same applications with the number of replicas that is equal to zero where the volumes are already prepared thanks to asynchronous replication.

The major strength of the Active-Passive approach is its operational ease. There are no issues associated with distributed consensus, split-brain situations, or simultaneous writing incidents. In case of any failure, the operation procedure is straightforward: first, verify the failure of Cluster A. Second, separate Cluster A from the network to eliminate any split-brain complications. Third, promote the storage of Cluster B to the read-write state. Fourth, increase the application pod replicas to the required level with the assistance of GitOps pipelines and lastly change the GSLB routing tables to Cluster B.

The downside of the Active-Passive approach is the ineffective use of resources. An organization is required to invest money, energy, and time into a full secondary OpenShift installation which is inactive on a daily basis.

Active-Active Setups

Active-Active method distributes customers’ requests between Cluster A and Cluster B at the same time during normal operations. If one of the clusters is down, the other one can take over and handle the workload from the inoperative cluster thus, making RTO almost equal to zero.

Deploying stateless applications has no complexities when using an Active-Active topology. In such case, the application can be deployed in both clusters, using GitOps, while the GSLB takes care of receiving requests in a round-robin manner, or using geo-proximity or least-connections methods.

Active-Active deployment of stateful applications brings many complications: when both clusters will be writing simultaneously to the same application data set, the standard block-level storage replication method would not be sufficient anymore. Simultaneous writing to the same storage block of two different clusters can immediately corrupt the file system.

To ensure the stateful Active-Active resilience, specific application architectures or globally distributed database engines are required:

• Distributed NoSQL Systems: Technologies including Apache Cassandra or MongoDB that operate over clusters using replication tools which are inherent to them.

• Distributed SQL Engines: Systems that include CockroachDB or YugabyteDB and handle distributed consensus using Raft or Paxos algorithms.

• Asynchronous Multi-Primary SQL: Database servers that utilize application-level conflict resolution tools designed to identify and rectify concurrent updates on similar records.

In these advanced architectures, OpenShift is only used as a computing manager and delegates to the data layer the responsibility for transactional correctness, data locality, and reconciliation. While Active-Active architecture aims at increasing the resource efficiency and providing seamless failover, it has a significant amount of networking overhead and latency and complicates debugging in case of the partition in the network. For engineers who want to acquire standard administration and mastering this high-availability architecture type, attending openshift redhat training on OnlineITGuru closes the gap from simple container orchestration to corporate cloud resilience.