Multi-Model LLM Inference Platform Based on Ollama, Docker, and Kubernetes

This article introduces the open-source project llm-inference-platform, which aims to address the challenge of enterprises efficiently deploying and managing multiple open-source large language models. The platform uses Ollama (inference engine), Docker (containerization), and Kubernetes (orchestration) to build a cloud-native architecture, featuring multi-model concurrency, elastic scaling, unified API, etc. It is suitable for scenarios such as private deployment and multi-tenancy, providing enterprises with production-ready LLM inference infrastructure.

With the explosive growth of open-source large language models, enterprises and developers are facing the challenge of how to efficiently deploy and manage multiple models. Different application scenarios require different model capabilities—from lightweight code completion to complex reasoning tasks—and a single model can hardly meet all needs. The llm-inference-platform project is designed to address this pain point.

The project uses a cloud-native technology stack to build a scalable multi-model inference service platform:

Ollama as the Inference Engine

Ollama simplifies the process of model downloading, configuration, and operation, supporting one-click deployment of open-source models such as Llama, Mistral, and CodeLlama.

Docker Containerization

Each model service is encapsulated in an independent Docker container, enabling environment and resource isolation, facilitating version management, dependency management, and horizontal scaling, with a highly consistent deployment process.

Kubernetes Orchestration and Scheduling

Leveraging Kubernetes' orchestration capabilities, it automatically handles scaling, load balancing, and fault recovery, adjusts the number of Pod replicas based on request volume, and automatically migrates services when nodes fail.

The platform has the following key features:

• Multi-Model Concurrency Support: Run multiple different LLMs simultaneously, with independent deployment and scaling, and flexible configuration of model combinations (e.g., code generation, dialogue, Embedding models).
• Elastic Scaling Capability: Based on Kubernetes HPA (Horizontal Pod Autoscaler) mechanism, it automatically adjusts the number of instances based on real-time load—scaling up during peaks to ensure responsiveness and scaling down during troughs to save costs.
• Unified API Interface: Provides an OpenAI-compatible API, supporting client libraries like OpenAI Python SDK and LangChain, allowing switching to self-hosted services without modifying code.
• Optimized Resource Configuration: Fine-grained resource configuration (CPU, memory, GPU quotas) ensures sufficient resources for critical services and avoids waste.

The deployment process is simple:

1. Prepare a Kubernetes cluster and GPU nodes (e.g., NVIDIA GPU)
2. Configure Helm Chart or use the provided Kubernetes YAML files
3. Define the model list and resource configuration
4. Execute the deployment command and wait for the service to be ready

After deployment, model services can be called via standard HTTP API or OpenAI-compatible SDK.

Suitable for various enterprise scenarios:

• Private Deployment: Enterprises with strict data security requirements can deploy LLM services in their intranets.
• Multi-Tenancy Environment: Provide isolated model instances for different teams/projects.
• A/B Testing: Run multiple model versions simultaneously to compare performance.
• Cost Optimization: Resource sharing and elastic scaling reduce inference costs.

GPU Resource Management

Optimize resource usage through model quantization (INT8/INT4), request batching, and multi-model GPU sharing.

Cold Start Problem

Alleviate the long model loading time by using preloading strategies and maintaining a minimum number of replicas.

Version Management

Supports rolling updates and blue-green deployment to ensure model upgrades do not affect online services.

The llm-inference-platform provides enterprises and teams with a practical solution for deploying multi-model LLM services in private environments. Combining Ollama's ease of use, Docker's portability, and Kubernetes' scalability, it builds production-ready inference infrastructure and is an open-source project worth attention for planning LLM private deployment.