Mount OCI Model Weights Dynamically into OpenShift Serving Runtimes

The machine learning production lifecycle works very differently from the production lifecycle of the traditional software engineering process. Classic cloud-native microservices typically involve a regular change in their business logic while the rigid dependencies remain steady. The deliverable in cloud-native microservices comprises a large artifact with a runtime, framework, and application code that adds up to a few mega-bytes at the most.

The process of engine deployment in machine learning makes this process reverse. The runtime, based on the framework, either Triton or vLLM, or specific Python-based inference tools, remains the same for a long time after it has been configured and tested with the proper hardware. In contrast to that, the weights of such models can vary from several gigabytes for some computer vision models to hundreds of gigabytes for some large language models. Compared to the serving engine, their weights are updated, retuned, and swapped much more often.

When thousands of extra gigabytes of frozen tensors are directly included into the product container, it creates the bottlenecks for the company. Each mini adjustment or hyperparameter change shall be followed by a lengthy run of the pipeline to generate, move, and store multiple container layers.

Considering model weights a major Open Container Initiative (OCI) artifact and loading them in an operating serving container at the moment of operation would be the best way to get rid of this architectural friction. This would allow for separating the implementation code from model weights and using an already existing enterprise container registry for distribution, thus making engineering teams design quicker and become more effective and secure in terms of AI inference solutions.

The problem of monolithic model containers

It is when these teams begin transitioning from using their trial notebooks to the Kubernetes or OpenShift infrastructure that they will choose the most straightforward path – bundling all the contents into a single container. The best way for platform engineers to sidestep such problems is by getting openshift training courses.

In theory, this approach is good enough, but it does not last long in the corporate world:

1. The Problem with Building and Networking

Creating a container image which involves a weight matrix of forty GB creates huge disk I/O and network overhead. Ordinary container makers face the daunting task of loading these huge files, forming intermediate layers, hashing every block, compressing the finished product, and transferring all of this through the company network to an external registry. Developers tasked with adjusting model variables can find themselves waiting for the container to be put together for up to an hour or two.

2. Container Registries and Storage Problems

Container registries handle immutability via the means of cryptographic digests and layer deduplication. Even slight changes in weight material mean a complete change in the layer. As a matter of fact, data science teams routinely have to send recurrent models every week leading to filling storages logjammed of repetitive frameworks on top of massive unique weight files.

3. Deployment Latency and Failed Auto-Scaling

Kubernetes and OpenShift automatically scale workloads according to network traffic, latency, and hardware utilization. If a service has a heavy traffic flow, the orchestrator can schedule extra pod replicas in cold worker nodes. If the model artifact is incorporated in the base image, the container runtime engine on this node has to download the entire image before scheduling the container. Downloading tens of gigabytes over an overloaded internal network will take a few minutes. Therefore, in the case of high demand this delay makes horizontal pod autoscaling useless and could result in lost requests or timeout messages before any newly created pods are in the healthy state.

4. Supply Chain Security and Vulnerability Scanners

Enterprise clusters need automated security scanners to check all incoming images against existing databases of vulnerabilities. When monolithic model images arrive to the registry, security scanners must unpack and index multiple layers of tens of gigabytes just to run scans for any operating system library packages and language dependencies.

Analyzing Storage Alternatives: Object Storage vs. OCI Registries

To avoid the use of hefty images, there is a unanimous understanding among teams that model weights need to be externalized. The common destination for externalized weights has been S3 compatible object storage since ancient times.

Even though object storage provides vast capacity and volume management separate from the process, it comes with certain operational challenges in the context of private companies:

Drawbacks of Object Storage

In order to retrieve a model from object storage during runtime, pods generally require either specific custom scripts, dedicated sidecars, or API calls at runtime. This requires the use of a credential management system in order to protect the credentials and ensure that each application namespace has either secrets, tokens, and/or IAM roles to access certain buckets of object storage.

Moreover, unless the storage client is capable of advanced multi-part range queries mapped to memory directly, the whole model must be first downloaded from a bucket to local storage in order for the runtime to commence. Since the same node may host several pods, each of them downloads its own copy of model weights from the bucket consuming available bandwidth and disk space.

The Benefits of the OCI Registry

The OCI registry operates by treating its data as standard content-addressed blocks. Nowadays, container registries are no longer confined to runtime images with root filesystems only; rather, they welcome any form of OCI artifacts.

When the weights of a model are stored as OCI artifacts in a registry, some key benefits are granted:

One unified infrastructure: Corporate infrastructure teams have already solved issues related to high availability, geographical replication, role-based access management, auditing, and high bandwidth inbound/outbound data transfer for their container registries. With models stored in the same registry, it is no longer necessary to worry about managing separate controls over multiple object store systems.

Built-in authentication and secrets: OpenShift includes registry image pulling secrets at service account and namespace levels. Pods obtain access to model artifacts through trusted credentials that secure software image pulling, which means that there is no need for custom S3 client programming and authentication tokens.

Intelligent layer caching: In cases when numerous different pods on one node utilize identical model layers or when one version of the model gets usage of tensor fragments that remain unchanged, layers must only be downloaded once by the local engine of the node where computing takes place. Following pods retrieve that content that is already in cache stored on the local disk, hence saving significant amounts of time.

Their model weights could be signed at an export stage by the teams and made sure that admission policies are working properly by OpenShift so that the nodes have only those weights executed that were handled by the procedures defined by model Good Governance.

Technology details of the ModelCar model

The model of using OCI artifacts as stored models has received fame in the industry as the ModelCar model which has found its place in KServe.

The ModelCar architecture does not try to use specialized storage drivers to connect to external repositories but rather relies on Kubernetes concepts.

The Separation of Concerns

This architecture relies on the combination of two complementary images for the purpose of implementing the prediction service:

The Generic Serving Runtime - which is an immutable, optimized image maintained by the platform/infrastructure team. It has all the necessary components: Linux distribution, Python runtime, machine learning frameworks, CUDA or any device drivers, and inference server binaries. This image remains unchanged over time and does not reference specific weight files and is shared among many workloads.
The ModelCar Artifact - which is a minimal container image containing just the model directory with weight files, tokenizer definitions, config files, and other info. It does not run any code, web servers, or advanced dependencies.

Lifecycle of Runtime: In a Stepwise Manner

Whenever a new service is launched or modification made on OpenShift, the following cycle takes place.

Job Scheduling and Pulling of Images: The OpenShift scheduler designates the pod to the node in question and CRI-O detects the image relating to ModelCar and downloads the layers from the private image repository. If a different pod had already downloaded the weights or if the weights were obtained from the earlier versions of the image having common layers, the download from the local disk would happen almost simultaneously.

Running Initialization Container: The ModelCar container runs as an initialization container and the only job of this container is to fill in the common pod volume with model files. Depending on the way of functioning, the initialization container can either upload weight files into the common volume or it may use the shared mounted method to upload the weights in a common directory accessible to the pod.

Init Container Completion and Pod Development: The init container has exited with a successful status after the validation of the file paths. The underlying volume now has the weight folder stored completely and uncompressed.

Main Runtime Activation: The generic serving container is started. Its configuration specifies the model directory argument to be pointing at the mount point of the shared volume. As a result, the serving engine then accesses the files, loads them into the memory of the CPU or GPU through standard calls, loads the tokenizers and configurations, and starts to listen on the specified network port.

Developing and Packaging the Model Weights in the Form of OCI Artifacts

In order to produce high-quality and enterprise-ready ModelCar images, it is essential to follow the packaging methodology in a proper way in line with security and performance.

Selection of the Base Layer

A ModelCar image needs to offer the minimal file system and shell capabilities in order for the standard runtime container to perform basic copying or linking in the init phase.

Minimalist Base Strategy: When it comes to Red Hat OpenShift ecosystems, it is customary to develop a solution based on a minimalistic enterprise image, such as UBI Micro. This image provides a lightweight environment—frequently smaller than 30 MB—that does not come with package managers, unnecessary daemons, or extra libraries. The minimal shell's capabilities are sufficient to meet the requirements imposed by the container lifecycle without creating vulnerabilities.

Limit of the Empty Image: Developers who are used to languages like Go or Rust may choose an empty image (or scratch) to lower their solution's size. However, an empty image cannot include even the shell programs. In addition to that, the scratch images cannot perform the scripts required for the proper functioning of the standard init containers of Kubernetes unless advanced shared-mount plugins are used, which run beyond the container's processes.

Model Payload Structure

The model weights have to be housed in a clean, standard directory structure within the image filesystem. Don’t go down overly deep, tool-specific build paths.

A production model folder typically includes:

First of all, all primary tensor weights; these might include so-called safe formats (for instance, safe tensors) or other forms of ONNX model.
Secondly, model architecture settings manifest file;
thirdly, tokenizer settings (parameters, vocabulary tables, merges);
fourthly, generation or inference parameters;
Fifthly, licensing and other technical information on the model.

The fact that the same structure is used across all models allows the generic serving engine to process arbitrary models automatically without the need for specific runtime or script options.

File Permissions and Security Context Constraints Management

By default, OpenShift has strong Security Context Constraints in place. Normally, root users are not allowed to run containers. In place of the root user, OpenShift picks a random User Identifier (UID) for the container process during runtime, which goes with a Group Identifier (GID) of root (group 0).

In the case where a model image is created in an unfortunate way with files owned by a specific root user with no read permission for the group, the Serving container with a random UID will receive permission errors from the operating system while trying to read the tensors.

To make sure things run without issues under OpenShift Security Context Constraints:

All weight files and directories under the model artifact must be owned by the root group (group zero).
The permissions must be set to ensure group-read and execution permissions for the directory structure (having non-root users access directories and read files).
Add a non-root user to the build instructions (such as the standard nobody UID) to avoid running the process with UID zero.

Integrating Runtime within OpenShift

To effectively link independent components of OpenShift, one needs to set up pod specifications to provide a connecting transit system for the init container and the running process.

Choosing Storage Medium: Disk vs. Shared Memory

When initializing the container passes one or several model files to the base serving container, there should be an intermediate volume used in between. Kubernetes gives several choices through empty directory volumes, their type of the volume is determined by the characteristics of the hardware of the cluster.

Host-Path Backed Ephemeral Volumes: usually the ordinary ephemeral volume is written to an ephemeral storage partition of the host node. Huge model files generate a large volume of information that needs to be transferred into the disk through the init process. But this is the most secure solution if the nodes have fast NVMe local disks available of required sizes.

Memory-Backed Volumes (RAM Disks): The volume uses memory-backed storage, which means it is supported by a tmpfs virtual memory filesystem in the host's RAM. Data transfer takes place at memory-bus speeds without the normal input-output contention of disks. While this is very efficient, the required amount of system memory on the worker nodes must be considerable to store both the original model files and the memory footprint of the executing process without the operating system triggering the kernel Out-Of-Memory (OOM) killer.

The Role of OpenShift AI and KServe

In modern OpenShift environments, managing low-level pod definitions manually for every model deployment is unnecessary. Red Hat OpenShift AI gives an abstraction of the above-described lifecycle through the built-in KServe component. In the case of engineers who are interested in gaining expertise in end-to-end workload orchestration ranging from basic cluster primitives to sophisticated AI Serving stacks, getting enrolled for an openshift full course can be very beneficial.

While configuring an inference service in OpenShift AI, a developer only needs to specify an OCI URI to their container registry instead of providing an S3 object store path. The underlying KServe controller is responsible for orchestration.

It creates relevant pod definitions.
It includes the ModelCar container as an initialization container.
It sets up the shared volume mounts across containers.
It dynamically sends the target file paths to the runtime engine that is in charge of the deployment of models (for instance, vLLM or OpenVINO).

This method of model deployment means that data scientists only specify the particular tag of the model that they wish to render. The infrastructure is responsible for mounting volumes, caching procedures, and lifecycle management.

CI/CD Pipeline Architecture: The Separation of Code and Weight

The shift from code to weight changes the process of continuous integration and continuous deployment (CI/CD). Instead of having one big pipeline that is launched as soon as the system introduces any changes to the model or app, the engineering team runs two separate, yet interrelated pipelines.

Pipeline 1: The Runtime Serving Pipeline

This pipeline has ownership by platform engineers, infrastructure teams, and machine learning operations specialists.

Trigger: Includes modifications to serving frameworks, dependency revisions, runtime performance fixes, or base operating system vulnerability advisories.
Process: The inference runtime (for example, optimized vLLM or Triton build) is compiled and verified, synthetic regression benchmarks are implemented against reference models, vulnerability enterprise scans are performed, final image signed, and published to the central repository with semantic version tags.
Frequency: Infrequent; usually updated monthly, quarterly, or set on the emergence of critical security patches.

Pipeline 2: The Model Artifact Pipeline

This pipeline is managed by data scientists, machine learning engineers, and automatic training workflows.

Trigger: Completion of training runs, fine-tuning jobs, parameter quantization, or safety assessments.
Process:

Applies copy, checks and checksum verification.
Copies weights, tokenizers and configurations to a workspace that contains the minimal UBI Micro base context.
Changes permissions to conform to OpenShift group ownership requirements.
Creates OCI artifact with Podman or automatic OpenShift Pipelines (Tekton) toolset.
Pushes the artifact to the enterprise registry (for example Red Hat Quay).
Cryptographic signature of layer digest is accomplished using enterprise signing keys.
Updates the GitOps repository, which monitors inference service configuration, to incorporate newly published image tags.

Frequency: high; updated with new models graduating from offline experiments and quality assurance milestones.

Operational Excellence and Day-2 Management

It is crucial to factor in the operational minutiae to manage the operation of disconnected model techniques seamlessly.

Registry Pruning and Retention Regulations

Since the model objects are of considerable size, a thriving data science group that develops several models a day can eat up all the storage space quickly, resulting in the utilization of hundreds of gigabytes of storage space on the registry. In this context, it is essential to create and implement a strong lifecycle and expiration tagging policy in the registry.

Utilizing a tag-purging system that removes untagged layers or models that have not been retrieved over a particular time period.
Providing safety tags or non-changeable locks for the models that have been promoted to staging, production, or long-term retention practice.
Employing storage systems of the registry which can apply deduplication across layers thus ensuring that only unique differences take up space if two teams publish new variations of a model that only diverge with minor differences in the underlying tensor values.

Host Node Image Garbage Collection

During the download of large images on the Kubernetes worker nodes, the disk space consumed by these layers may quickly reach node disk pressure limits. Once the node disk capacity exceeds high watermarks (usually eighty to eighty-five percent), Kubernetes starts an aggressive node image garbage collection, which deletes unused container images.

If the disk is small, the node will download a forty-gigabyte ModelCar, execute the model, then shrink the pod and immediately remove all image layers due to disk pressure. When requests come back, the pod will be scheduled, and the node will have to download the whole artifact from the registry, bypassing the benefit of locally cached layers.

How to address that:

Provision worker node root file systems or separate container storage volumes with sufficient space for concurrent caching of model layers along with regular operational padding.
Configure the kubelet with proper settings for garbage collection threshold to match the size of the models that your organization uses.
Use special inference pools of worker nodes that run the same model in order to increase cache hit rate upon pod restarts.

The configuration of storage on the worker nodes, adjusting garbage collection thresholds for kubelet, and implementation of strict security context constraints form skills that are proven by an openshift certification course.

WEEKEND SPECIAL - UPTO 60% OFF