LLM-D-Lab: A Complete Solution for Automating Deployment of Large Model Inference Experiment Environments on OpenShift

LLM-D-Lab is an automated solution for large model inference experiment environments designed specifically for the OpenShift/OKD platform, aiming to address the challenges of efficient and reproducible deployment of enterprise-level large language model inference systems. The project uses GitOps to automate the configuration of GPU worker node pools, core operation and maintenance components, observability systems, and traffic control, providing out-of-the-box experimental workloads. Target users include performance engineers, platform engineers, solution architects, and researchers. Currently, it supports two major cloud platforms: AWS and IBM Cloud.

LLM-D-Lab is a supporting experimental environment tool for LLM-D, an open-source large model distributed inference project. Target users include: performance engineers who need to run LLM-D and OpenShift AI benchmark tests, platform engineers/SREs building scalable LLM service infrastructure, architects prototyping LLM solutions, and researchers verifying distributed inference engines. The project currently supports AWS and IBM Cloud and plans to expand to more cloud providers.

Infrastructure Automation

Achieve automatic scaling of GPU nodes through MachineSet, MachineAutoscaler, and ClusterAutoscaler, and elastically adjust resources based on load changes to save costs.

Core Operation and Maintenance Components

• NVIDIA GPU Operator: Configure GPU drivers and monitoring components
• Node Feature Discovery (NFD): Detect node hardware features and label them
• Descheduler: Optimize pod distribution
• KEDA: Event-driven autoscaling

Network and API Gateway

• Gateway API: Next-generation service network API
• Kuadrant: Multi-cluster traffic management and API governance
• Authorino: Kubernetes-native authentication and authorization
• cert-manager: Automated TLS certificate management

Observability System

• Grafana: Monitoring dashboards
• NetObserv: eBPF network traffic observation
• LokiStack: Log aggregation

Experimental Workloads

Provide KServe LLMInferenceService examples and KV cache routing configurations, supporting precise prefix cache-aware experiments.

LLM-D-Lab adopts the GitOps-first methodology, with all configurations managed via ArgoCD to achieve declarative infrastructure management. Core advantages:

• Version control: Configurations are stored in Git repositories, with traceable change history
• Reproducibility: Versioned manifests can reproduce consistent configurations across different environments
• Automated synchronization: ArgoCD continuously monitors and synchronizes cluster states
• Approval workflow: Implement change review through Git branches and merge requests

The project avoids local scripts, prioritizes declarative manifests and Kubernetes control loops, reduces tool dependencies, and improves standardization and portability.

Deployment process using AWS as an example:

1. Clone the repository and configure the GitOps root application: Modify overlays/aws/root-app.yaml to fill in cluster API identifiers, regions, and other information. It is recommended to fork the repository to avoid relying on upstream status.
2. Fill in secrets configuration: Create actual secrets files based on the 99-*.example.yaml template.
3. Deploy the root application: Execute oc apply -k overlays/aws/ to trigger ArgoCD to create sub-applications.
4. Wait for readiness: Check the status via OpenShift WebUI or command line. Initial setup requires waiting for node scaling.

Note: Initial deployment may take a long time, especially during cluster scaling.

LLM-D-Lab's design follows three key principles:

• Modularity and Scalability: Support user-customized configurations through the Kustomize overlays mechanism without modifying core manifests
• Cloud-native First: Fully leverage the capabilities of Kubernetes, OpenShift, and the Operator pattern, without relying on platform-specific scripts
• Experiment-oriented: Provide standardized sample workloads to allow researchers to quickly start experiments and reduce environment setup time

These principles ensure the flexibility and practicality of the solution.

Known Limitations

• Incomplete uninstallation support: OLM-managed Operators need manual cleanup
• Single Node Cluster (SNO) considerations: The master node does not host user workloads; it is recommended to prepare worker nodes in advance
• RHOAI and upstream LLM-D components: Need manual deployment due to compatibility issues

Future Plans

• Improve IBM Cloud overlay coverage
• Support RWX storage classes
• Optimize CertManager, Kuadrant, and Authorino configurations
• Add more Grafana dashboards
• Implement multi-tenancy and concurrent experiment management (Tekton/Kueue)
• Support HyperShift managed clusters and multi-cluster management
• Provide more sample workloads

The project will continue to iterate to enhance feature coverage and user experience.

LLM-D-Lab represents a modern approach to AI experiment environment management: it implements infrastructure as code via GitOps, automates component lifecycles using the Operator pattern, and ensures scalability and portability through a cloud-native architecture. This solution not only simplifies the complexity of setting up large model inference experiment environments on the OpenShift platform but also establishes reproducible, auditable, and collaborative experiment workflows, which have important reference value for relevant teams.