llm-infer: A Unified Multi-Backend Large Language Model Inference Server

With the rapid development of Large Language Model (LLM) technology, the problem of fragmented deployment in production environments has become prominent. The llm-infer project provides a unified inference server architecture that supports native PyTorch/Transformers, vLLM, and Ollama backends, simplifying multi-model deployment and management while maintaining a consistent interface experience, helping developers flexibly choose backend solutions.

Current mainstream LLM inference solutions each have their pros and cons:

• Native PyTorch/Transformers: High flexibility, easy to debug and customize, but insufficient high-concurrency performance—suitable for research prototype stages;
• vLLM: High throughput and GPU utilization, ideal for large-scale production deployment, but complex to configure;
• Ollama: Simple and easy to use, one-click local operation, suitable for quick verification by individual developers, but limited enterprise-level features. Development teams face a dilemma in choosing, and llm-infer solves this problem through an abstract unified architecture.

Adopts a layered architecture with decoupled interface and implementation layers:

1. Unified API Layer: Standardized RESTful API ensures consistent interfaces across different backends, bringing application portability, simplified operation and maintenance, and convenient A/B testing;
2. Backend Adapters: Corresponding to each backend, handling model loading, request format conversion, response encapsulation, and error retries;
3. Client SDK: Supports connection pooling, load balancing, automatic retries, streaming responses, and unified authentication configuration.

• Dynamic Backend Switching: Switch backends at runtime based on load (e.g., use native during low peaks, vLLM during high peaks);
• Model Hot Reload: Prepare new models in the background for seamless switching without service interruption;
• Multi-Model Concurrency: A single instance serves multiple models simultaneously, each configurable with different backends;
• Intelligent Request Routing: Route to the optimal backend based on features like input length and priority.

• Development Environment: Use the Ollama backend for quick startup with minimal configuration;
• Testing Environment: Use the native backend for easy debugging and log tracking;
• Production Environment: Use the vLLM backend to maximize hardware utilization and support high concurrency;
• Hybrid Deployment: Choose based on task characteristics—e.g., vLLM for real-time applications, native for batch processing.

Performance Optimization: Asynchronous IO for concurrency, backend connection pooling, response caching, batch processing to merge small requests; Ecosystem Integration: Compatible with OpenAI API format, mainstream model formats (HuggingFace, GGUF), integration with LangChain/LlamaIndex, and Prometheus metric export.

llm-infer represents an important direction in the LLM deployment field: maintaining flexibility while reducing complexity. Its "backend-agnostic" architecture will play a greater role as LLM applications expand. It is recommended that teams planning or optimizing LLM infrastructure conduct an in-depth evaluation of this open-source solution.