on-prem-llm-stack: A Practical Guide to Enterprise Private LLM Deployment

This article introduces the GitHub project on-prem-llm-stack (author: ashmarev, published on 2026-05-29), which provides a local LLM deployment solution for enterprise compliance scenarios. The core goal is to prove that "data not leaving the country" is no longer an obstacle to LLM deployment, and local inference can be achieved even with a limited budget. The content covers hardware selection (from consumer to server GPUs), vLLM inference optimization, and planning for enterprise-grade features (proxy, authentication, etc.).

With the popularization of LLMs in enterprise applications, data security and compliance have become key issues. Organizations face constraints: 1. Data not leaving the country (e.g., Russian 187-FZ and other regulations); 2. Model controllability (version, update rhythm); 3. Audit tracking (operation logs). Cloud services cannot meet these needs, so private deployment has become inevitable.

The on-prem-llm-stack is implemented in four phases:

Phase	Module	Description	Status
01	inference	Local inference (model selection, hardware configuration, vLLM deployment)	✅ Ready
02	proxy	Routing, rate limiting, logging	🔜 Coming
03	agents	Agent architecture, tool calling, auditing	🔜 Coming
04	auth	SSO, RBAC, auditing	🔜 Coming
Currently, only the first phase is completed, laying the foundation for subsequent enterprise features.

The project tested multiple GPU configurations:

1. Entry-level (10k rubles): RTX2080Ti 11GB → 4B-AWQ model, vLLM supports 8 concurrency, 572 tokens/s, p99 TTFT 0.7s (suitable for proof of concept);

2. Mid-range (80k rubles): RTX4070 12GB → 4B-FP8 model, 32 concurrency, 1574 tokens/s, p99 TTFT 93ms (production for small and medium teams);

3. Enterprise-level (1.5 million rubles): A100 80GB → 4B-FP8 model, enterprise-grade stability (large-scale production).

Consumer GPUs (RTX series) are low-cost but lack ECC memory; server-grade GPUs (A100/H100) are stable but expensive.

The project uses the vLLM engine, leveraging PagedAttention technology to improve GPU memory utilization, concurrency, and throughput. Key findings: "LLMs can be deployed within any budget boundary", and "no infrastructure" is no longer an excuse. Quantization technologies (AWQ4bit, FP8) reduce hardware requirements: 4bit quantization reduces memory usage by 75%, and FP8 balances quality and performance.

Phase 2 (Proxy): Intelligent routing, rate limiting, logging, load balancing; Phase 3 (Agents): Tool calling, ReAct mode, audit tracking, security sandbox; Phase4 (Auth): SSO integration, RBAC, audit logs, key management.

Applicable scenarios:

1. Financial industry: Sensitive customer data processing, compliance requirements, audit records;

2. Government and critical infrastructure: High security level, network isolation, system availability;

3. Healthcare: HIPAA/GDPR compliance, data privacy, controllable model behavior;

4. Enterprise IP protection: Core intellectual property, trade secrets, internal knowledge base retrieval.

1. Progressive deployment: Phased implementation reduces risk, and each phase can be verified independently;

2. Cost-effectiveness: Different budget plans adapt to different scenarios;

3. Quantization optimization: AWQ/FP8 balances performance and hardware cost;

4. Compliance first: Consider requirements like data not leaving the country and auditing from the initial design stage.