RunPod worker-vllm: A Production-Grade Large Model Service Endpoint Deployment Solution Based on vLLM

RunPod's officially open-sourced worker-vllm template combines the high-performance vLLM inference engine with Serverless GPU infrastructure. It provides OpenAI-compatible APIs, multiple quantization methods, and flexible environment variable configurations, simplifying the process of building production-grade large model service endpoints.

With the rapid development of LLMs, efficiently deploying inference services in production environments has become a challenge: traditional methods are complex to configure and high in cost, while vLLM, though high-performing, has a steep deployment learning curve. RunPod launched the worker-vllm template to lower the deployment barrier and provide out-of-the-box OpenAI-compatible LLM endpoints.

worker-vllm is an officially maintained Serverless Worker template by RunPod, used to deploy vLLM-based LLM service endpoints. It is based on vLLM version 0.20.2, requires CUDA ≥13.0, uses Docker containerized deployment, and provides pre-built images (runpod/worker-v1-vllm:).

Two Deployment Modes

Option 1 (Recommended): Pre-built Image Use the pre-built image directly; configure environment variables to start, supporting any Hugging Face-compatible model. Option 2: Custom Image Package the model into the image via Docker build parameters, supporting offline/compliance scenarios, and allowing selection of vLLM nightly versions.

Environment Variable Configuration

Covers multiple dimensions:

• Model Configuration: MODEL_NAME, MAX_MODEL_LEN, QUANTIZATION (AWQ/GPTQ, etc.)
• Hardware Configuration: TENSOR_PARALLEL_SIZE, GPU_MEMORY_UTILIZATION
• Inference Optimization: MAX_NUM_SEQS, ENABLE_CHUNKED_PREFILL
• API Configuration: CUSTOM_CHAT_TEMPLATE, ENABLE_AUTO_TOOL_CHOICE Supports automatic discovery of vLLM AsyncEngineArgs fields (converted to environment variables in uppercase).

worker-vllm provides OpenAI-compatible interfaces, enabling seamless migration of existing client code. Supported endpoints include:

• Chat Completions (streaming output)
• Models
• Responses API
• Anthropic Messages API Multi-protocol support enhances the solution's versatility, adapting to downstream applications of different SDKs.

Inherits vLLM's extensive model support: mainstream open-source models like Llama, Mistral, Qwen, ChatGLM, etc.

• Private/gated models: Pass the access token via HF_TOKEN; custom images can use Docker secrets to protect the token.
• Supporting tools: RunPod provides a vLLM load balancer, supporting a highly available multi-instance architecture.

Applicable to multiple scenarios:

• AI Application Backend: Provides stable inference for chatbots, content generation, etc. Serverless pay-as-you-go avoids resource waste.
• Development and Testing: Quickly set up test endpoints to verify model effects and debug prompts.
• Model Comparison: Switch configurations to compare performance of different models, aiding selection.
• Private Deployment: Provides enterprises with a way to deploy open-source models on private clouds, ensuring data sovereignty.

worker-vllm lowers the barrier to large model deployment. Combining vLLM's performance with Serverless elasticity, it provides a production-ready, easy-to-use, and cost-controllable solution. Outlook: Future support for multimodal inference, finer-grained quantization, and intelligent scaling strategies is expected. Teams needing to quickly launch LLM services are recommended to evaluate this solution.