Private LLM SaaS: A Self-Hosted Large Language Model Backend Solution Based on LiteLLM and Ollama

Introducing an open-source self-hosted LLM SaaS backend project that supports LiteLLM and Ollama, offering secure API endpoints, user authentication, team key management, and fully containerized deployment.

With the rapid development of Large Language Model (LLM) technology, more and more enterprises and developers want to deploy and run these models in local or private environments. However, directly using open-source models often faces challenges such as complex deployment, inconsistent APIs, and lack of user management.

Private LLM SaaS emerged to address these issues, providing a complete self-hosted backend solution that allows users to build an OpenAI API-like service experience on their own infrastructure while maintaining full control over data and models.

The project adopts a modular design, with core components including:

LiteLLM is a unified LLM calling library that supports over 100 different LLM providers and models. By integrating LiteLLM, Private LLM SaaS can:

• Unify API interfaces for different models
• Support multiple backends such as OpenAI, Anthropic, Cohere, and local models
• Implement model routing and load balancing
• Provide standardized request/response formats

Ollama is one of the most popular local LLM running tools currently, supporting various open-source models like Llama 2, Mistral, CodeLlama, etc. The project deeply integrates Ollama, enabling:

• No need for complex CUDA or machine learning environment configuration
• One-click pulling and running of open-source models
• Support for model quantization to reduce VRAM usage
• Local inference, ensuring data does not leave the environment

The project has a built-in complete user authentication and authorization mechanism:

• User Authentication: Supports API Key and JWT Token authentication
• Team Management: Supports multi-team isolation, with each team having an independent key space
• Permission Control: Fine-grained access control, allowing restrictions on model access and usage quotas
• Audit Logs: Complete request log records for compliance review

The entire backend is fully containerized based on Docker, providing:

• Docker Compose: One-click startup of the complete service stack
• Kubernetes Support: Provides Helm Chart for production environment deployment
• Environment Isolation: Complete isolation of development, testing, and production environments
• Scalability: Supports horizontal scaling to handle high-concurrency scenarios

The project provides RESTful interfaces compatible with the OpenAI API, including:

• /v1/chat/completions - Chat Completions
• /v1/completions - Text Completions
• /v1/embeddings - Text Embeddings
• /v1/models - Model List

This means you can directly use OpenAI's SDK or any compatible toolchain to access self-hosted models.