Unified Ceph Storage Across Every Hybrid OpenShift Cloud
Last updated on Aug 31, 2026
Contemporary corporate IT occurs in the setting of private datacenters, co-location sites, and various types of public clouds. Even though container orchestration platforms such as Kubernetes have provided a single technology for the packaging and deployment of computing processes, historically, data persistence has been quite a challenge.
Each of the providers of cloud services and hardware manufacturers establishes its rules of storage and distinct APIs, performance characteristics, and lifetime of systems. A program deployed on a cloud service such as Amazon Web Services requires some techniques of block and object storage. On the other hand, it will be required to use some other technologies when working with the deployment of the same program on a physical hardware server.
The difference between storage systems contradicts the premise of cloud-native computing, claiming once written, a program can be used anywhere.
The Red Hat OpenShift Data Foundation (ODF), formerly known as Red Hat OpenShift Container Storage, helps eliminate this existing gap in the system infrastructure. The structured openshift training offered to engineers and administrators prepares them for enterprise workloads by enabling their practical experience in deploying software-defined storage solutions that operate within the OpenShift environment.
Architectural Aspects: Ceph, Rook, and NooBaa
To understand what makes OpenShift Data Foundation operational in heterogeneous environments, one has to study the three constituent components of its technology stack: Ceph being the distributed data engine, Rook—the Kubernetes-based storage broker, and NooBaa—the Multicloud Object Gateway.
Ceph: The Distributed Storage System
At the base level of the ODF solution Ceph is situated, a massively scalable open-source platform. Ceph does not need any centralized server for tracking the data – the platform rather uses Controlled Replication Under Scalable Hashing algorithm to locate the data.
Once a data block, file chunk, or object comes into the storage system, CRUSH determines where to place it in the cluster based on the parameters for failure domains that the user has selected. Some of the failure domains that may be configured include disks, hosts, racks, power units, or cloud availability zones. This kind of arrangement can provide several advantages when it comes to cloud-native storage:
Autonomous Self-Healing and Rebalancing: In the event of failure of storage node/disk, other nodes involved in the working process sense the situation and start calculating the placement groups that have less than required number of replicas. Nothing is needed to be done from the side of the administrator as the respective device is able to automatically find out where the replicas need to be placed.
Uniform Scalability: This means that storing capacity and efficiency grow in accordance with installation of more disks/nodes. The more disks/nodes are added, the higher is the capacity of the system.
Built-in Support for Multiple Protocols: Ceph provides block storage, file system storage, and object storage by means of RADOS, CephFS, and RGW correspondingly.
Rook: The Cloud-Native Storage Orchestrator
Despite Ceph's provision of data resilience and storage protocols, in the past, there was a degree of operational complexity associated with running a distributed storage system within a container orchestration framework. In this regard, Rook provides an answer to this problem as an open-source Kubernetes Operator which automates Ceph cluster deployment, bootstrapping, configuration, provisioning, scaling, upgrading, and recovery.
Rook converts Kubernetes declarative specifications to low-level operations on Ceph clusters. It uses custom resources that represent storage clusters, block pools, file systems, and object storage within the Kubernetes control framework.
Rook Operator monitors the declarative states without stopping. If any storage node suffers from hardware failure, the Rook orchestrates the lifecycle of replacement; if an administrator changes the version tag on the OpenShift Data Foundation Operator, Rook manages the rolling zero-downtime upgrade of Ceph Monitor, Manager, Metadata Server, and Object Storage Daemens, while the persistent volume is still available.
NooBaa: What is Multicloud Object Gateway?
The third architectural pillar of the solution is referred to as NooBaa, which is the technology powering ODF Multicloud Object Gateway (MCG). NooBaa functions in such a way that object storage consumption is abstracted by exposing S3 type compatible API to applications and separating client interface from the physical backing store.
It is also worth mentioning that NooBaa can direct object data either to Ceph supported of internal storage pools or federate and tier data between different cloud object storage services, such as Amazon Simple Storage Service (S3), Google Cloud Storage, Microsoft Azure Blob Storage and IBM Cloud Object Storage.
With NooBaa, platform engineers are able to define place policies, encrypt data on the client side before sending it to public clouds, mirror the objects between different cloud regions in real time and also store cold data on a low-cost archive storage without breaking the contracts related to the functionality of the application API.
Three Type of Storage in OpenShift
Enterprise applications operating on containers generate IO characteristics that vary significantly. For instance, a transactional relational database calls for low latency block volumes to avoid having multiple containers accessing it. On the other hand, the CI pipeline or content management system needs multiple containers to write/read on the same content at the same time. In addition to these, an AI model training requires object storage that can be expanded and provide access through HTTP. All these storage requirements are met by the ODF solution based on Kubernetes architecture and features.
Block Storage (Ceph RBD / ReadWriteOnce)
Block storage refers to the basic storage that directly connects to a particular computing resource on the server. In OpenShift, block storage is provided through the Ceph Container Storage Interface (CSI) driver as persistent volumes configured with ReadWriteOnce (RWO) policy.
Whenever a pod requires an RWO persistent volume claim from the ODF block storage class, the CSI provisioner brings an image from the Ceph block pool, maps that image to a block device, sets up that image with a file system (e.g. ext4 or XFS), and mounts it into the pod's container namespace.
Ceph RBD storage is able to provide optimal performance in transactional workloads, since this technology is capable of avoiding the overhead associated with high-level file sharing protocols and writing directly to the objects within the RADOS. Thus, it is the main type of storage for the following:
Transactional database applications: Relational database management systems, like PostgreSQL, MySQL, Microsoft SQL Server etc.; enterprise deployments of Oracle Database running inside containers.
Messaging and logging applications: Workloads using technologies like Apache Kafka, RabbitMQ, Apache Pulsar requiring sequential write performance and deterministic disk flushing.
OpenShift Virtualization: VMs running on OpenShift through KubeVirt with ODF block storage offering direct virtual machine disk image storage and execution of live migrations across nodes without any unmounting delays.
Shared file storage (CephFS)
Many microservice architectures and batch distributed processes require shared file access. In this regard, many application instances located on various worker nodes perform reading and writing operations in parallel. The problem is solved in standard cloud services through separate managed NFS services that allow setting up the access outside of the Kubernetes control plane.
ODF provides native shared file storage by means of CephFS implemented through Ceph-CSI that provides ReadWriteMany (RWX) mode. CephFS works by distributing the files across RADOS objects and utilizing dedicated Ceph metadata server (MDS) capable of maintaining the hierarchical directory structure, file attributes, ACLs, and file locking semantics.
CI/CD Caching: Build engines like OpenShift Pipelines (Tekton) and Jenkins in which build artifacts, maven caches and container layer caches are shared between different nodes that run various ephemeral builder pods.
It takes specialized operational expertise to configure multi-node storage access, CephFS metadata servers, and access permissions. Attending a professional open shift training equips cloud architects and admins with the required knowledge, including persistent volume provisioning and complex container storage processes.
Content Management and Document Storage: Farm of web applications such as WordPress, Drupal or proprietary document management systems in which many application copies access a shared storage of materials and modify them.
AI/ML Model Development and Sharing: Using Jupyter notebooks and data science environments that allow teams to share the same training data set, exploratory notebooks and model checkpoints among themselves.
The Object Storage Solution Compatible with S3 (NooBaa/Object Bucket Claims)
Standard storage interfaces are based on the assumption that any application runs on either the file system (or in its local network) or on the same kernel. On the contrary, modern software architectures built for cloud computing increasingly prefer storing objects via simple REST APIs supported by HTTP (and HTTPS) protocols.
The ODF is a game changer regarding object storage because it enables developers to use Object Bucket Claims to get buckets in an OpenShift environment declaratively. In simple terms, once a developer creates the Object Bucket Claim (OBC), the Multicloud Object Gateway creates an S3-compliant bucket and produces a ConfigMap and Secret in Kubernetes, thus providing an access key, secret key, and endpoint. The application pod therefore obtains all the necessary information and can authenticate on the S3 endpoint without any previous infrastructure set-up.
This storage type ensures:
Cloud-Native Application Architectures: Applications built using S3 APIs, which allow them to upload media, process asynchronous attachments, or track telemetry data.
Data Lakes and Analytics: A method used to store unstructured, semi-structured, or structured datasets that can be accessed by various analytical querying engines.
OpenShift Cluster Services: Used to support the internal OpenShift container image registry, hold OpenShift logging archives, and serve as a backup destination for APIs related to Data Protection and Velero.
Deployment Topologies: Internal vs. External Modes
OpenShift Data Foundation supports two main deployment architectures according to the size of the infrastructure, performance needs and limitations, structure of operation teams, and financial abilities:
Internal Mode and External Mode
In Internal Mode (also known as hyper-converged deployment), OpenShift Data Foundation runs completely within the cluster of the OpenShift Container Platform. The Ceph storage daemons (such as Monitors, Managers, OSDs, and Metadata Servers) operate as containerized pods in OpenShift worker nodes, utilizing either local physical NVMe/SATA SSDs or cloud block volumes that are attached dynamically (e.g. Amazon Elastic Block Store, Azure Managed Disks, Google Persistent Disks).
To achieve high dependability and reliability in Internal Mode, usually the organization needs to have at least three worker nodes that are used as dedicated storage nodes and are located in a few different availability zones or failure domains.
The nodes in this technique must be tagged and damaged to stop the workload of computing from affecting the resources that are necessary for running Ceph daemons. This is because Ceph has to have CPU and memory specifically allocated for Object Storage Daemons (OSD) for it to be able to manage the processes related to data placement and transfer itself at a high load.
The Internal Mode guarantees many benefits for using it in either small or medium enterprise setups, remote edge locations, and one-off development installations:
Sound Management: The infrastructure for storage is managed together with the computing workload through the OpenShift console and managed through OperatorHub.
No Additional Infrastructure Costs: There is no need to buy, license or protect any external storage tool; ODF joins together and makes the use of all the raw storage disks in the respective cluster nodes possible.
Self-deployment: The people working for the platform can get the enterprise storage system ready to work in just several minutes, as the process is automated.
External Mode
Ceph storage cluster and OpenShift Container Platform cluster are independent of each other in External Mode as both components are managed and deployed independently. The Red Hat Ceph Storage is installed on separate, physical bare-metal servers outside of the OpenShift Container Platform cluster.
The OpenShift Container Platform runs ODF endpoint layer which uses CSI plugins and NooBaa gateways to connect the workloads running in containers to the Ceph storage cluster.
External Mode is mainly preferred in large-scale organizations and in centralized datacenter environments where architecture is a key factor:
Compute Unit and Storage can be scaled individually: Companies can enlarge their storage volumes that span from terabytes to dozens of storage drives but have no necessity to increase the number of OpenShift compute nodes, and vice-versa.
A single Red Hat storage cluster may perform as block storage, file storage and object storage at the same time for numerous OpenShift clusters optimizing data center hardware utilization.
Dedicated resource segregation: the processes involving storage capacity use do not affect performance of the containers used to run important applications.
OpenShift clusters can be changed independently of the permanent data level of the data.
Understanding Hybrid Cloud Strategy
Infrastructure diversity is an important characteristic of enterprise computing today. Organizations may use VMware-based datacenters on-premises while using AWS for running customer services, Google Cloud for AI workloads, and Azure for analytics.
Using hybrid cloud was complicated in terms of operations since every cloud environment had its own storage challenges:
Storage API Compatibility: Developers had to develop cloud-specific code as applications required different storage technologies, e.g. an application required S3 in AWS, Blob in Azure, and NFS on-premises.
Dynamic Binding Disparity: Different storage types had different parameters such as mounting speed, input/output operations per second, allocation sizes, etc.
Data Gravity and Lock-in: When moving data-driven workloads between private and public clouds, organizations had to transform snapshots, reconfigure storage configurations, and redesign data access processes.
OpenShift Data Foundation solves these issues by being a universal software-defined abstraction layer across all cloud platforms.
Uniform APIs and consistency in operations
Once OpenShift and ODF have been adopted across a hybrid infrastructure, engineers can create Kubernetes PersistentVolumeClaims with ODF storage classes no matter where the cluster runs—whether on bare-metal, OpenStack, VMware, AWS, Azure, GCP, or IBM Cloud.
Block storage will be requested via Ceph RBD, shared file storage via CephFS, and object storage via the Multicloud Object Gateway.
Since the behavior of storage, locking, snapshots, and API remains the same at every level, applications can reliably be transferred to any public cloud production environment without changing any lines of code.
Cross-cloud Object Federation and Data Mobility
With Multicloud Object Gateway, data mobility through hybrid clouds is revolutionized, says Oracle Data Flyweight (ODF). NooBaa S3 endpoint serves as the main entry point for communication between applications and ODF. Team members setting up NooBaa have the option of customizing their own data policies:
Data Tiering and Archiving: Hot data from the applications is written to high-speed local Ceph NVMe storage pools for processing. With time, cold objects will be automatically migrated by NooBaa into cost-effective public cloud tiers (AWS S3 Standard-Infrequent Access/Azure Cool Blob) without any change in the original object path that is available to applications.
Real-time Data Mirroring: With an Object Bucket Claim set-up, applications have the ability to do a real-time mirroring of their objects to various cloud providers at the same time. An application's upload of a file triggers NooBaa to asynchronously replicate the object in two places – the local Ceph store and AWS S3 bucket. This makes sure that there is no data lock-in in cloud providers and gives immediate business continuity even if the whole cloud goes down.
Cost Optimisation on Egress: One of the issues with hybrid cloud solutions is that egress charges are often high because cloud applications are constantly pulling datasets from internal data centers. NooBaa caching and namespaces allow caching of data in the cloud cluster and local read after the first fetch, reducing transit charges.
Security, Governance and Encryption for Enterprise
The performance of enterprise functions using private and public cloud networks requires a high level of security standards. OpenShift Data Foundation incorporates secure mechanisms into each layer of the storage system.
Encryption at Rest
Data stored on physical storage media must be protected from both unauthorized access and access from the lower-level hypervisor. The ODF offers full encryption at rest using different levels:
OSD Encryption for the Whole Cluster: ODF has the capability to set up LUKS encryption automatically at the time of setting up clusters of storage by turning on LUKS encryption for all the object storage daemons available. Each data that is written on the disk is encrypted using AES-256 while moving towards the physical blocks to ensure that data is safe if the physical disks are lost.
PV Encryption per Tenant: Even though the cluster-wide encryption provides protection of the physical disks, companies operating multi-tenant environments need different keys for different applications. This is where the ODF Advanced comes in, allowing the companies to encrypt persistent volumes with the tenant’s keys. Thus, even if the cluster admin accesses the node maintaining the persistent unit's storage, they won't be able to read the data kept for a particular tenant without a proper security clearance.
Interoperability of Key Management
Carrying out key management in a cluster might compromise security. ODF uses enterprise Key Management Systems (KMSs), which are linked with ODF via different protocols like the Key Management Interoperability Protocol (KMIP), and has its own links with HashiCorp Vault.
In such setups, the encryption keys and the master secret are not stored in the OpenShift cluster but in a vault that manages the hardware security module itself.
Encryption in Transit
The data flowing over either the internal cluster network or the wide-area networks must be guarded against packet sniffing. ODF provides full encryption in transit:
Ceph Messenger Protocol 2 (msgr2): ODF encrypts all communications between Ceph daemons (monitors, managers, OSDs) and between CSI drivers at OpenShift worker nodes and storage nodes via AES-GCM encryption implemented in the Ceph protocol.
Object Access TLS end-points: Multicloud Object Gateway and Ceph RADOS Gateway expose their S3 endpoints that are protected through modern Transport Layer Security (TLS 1.3) technology that requires mutual authentication and guarantees messages encryption during RESTful interaction.
To Summarize: Unifying Data Services
The enterprise path towards cloud-native architecture cannot be achieved without independent data storage from the disjointed, vendor-specific, and proprietary hardware interfaces. While Kubernetes addressed the application compute cycle, Red Hat OpenShift Data Foundation tackles the cloud-native data cycle.
By bringing together Ceph’s distributed resilience, Rook’s autonomous orchestration, and NooBaa’s hybrid flexibility, the ODF actually builds a platform-independent storage layer in the real sense of this concept.
The Open Data Format (ODF) offers high Input/Output Operations per Second (IOPS), block storage for transactional databases, POSIX-compliant file storage for collaborative applications, and S3-style object storage on any multi-cloud platform. To master the practical application of the above-mentioned enterprise architectures, a complete openshift course from Online IT Guru is the best option for DevOps engineers and system administrators.