llm-pool: FastAPI-based LLM Inference Pooling Service Supporting Hybrid Local and Remote Deployment

Core Introduction llm-pool is a FastAPI-built LLM inference pooling service supporting mixed deployment of local models and OpenAI-compatible remote APIs. It offers scheduling management, replica control, metrics monitoring, and admin API functions, ideal for enterprise scenarios requiring unified management of multiple LLM backends.

Source Info

• Maintainer: Bobcat
• Platform: GitHub
• Release Time: 2026-06-09
• Repository Link: https://github.com/Bobcat/llm-pool

Key Challenges

1. Resource Fragmentation: Organizations use diverse LLM resources (local open-source models like Llama/Qwen, third-party APIs like OpenAI/Azure OpenAI, in-house models) without unified management.
2. Load Imbalance: Peak overload on some models while others are idle, lacking dynamic scheduling.
3. Observability Gaps: No unified metrics for call volume, response time, error rate, or cost distribution.
4. Scalability Limits: Adding new backends requires code changes and redeployment.

llm-pool solves these by integrating scattered resources into a manageable, monitorable, scalable service.

FastAPI Foundation

• High performance (Starlette/uvloop), async-native, type-safe, auto-generated OpenAPI docs.

Pool Model

• Local Backends: llama.cpp, vLLM, TGI, or custom OpenAI-compatible local services.
• Remote Backends: OpenAI, Azure OpenAI, Anthropic, or other compatible third-party APIs.

Scheduling Policies

• Round Robin, Weighted Round Robin, Least Connections, Response Time Aware, and custom plugins (cost-based, content-based routing).

Replica Management

• Horizontal scaling, failover, health checks, graceful shutdown.

Metrics Monitoring

• Request-level (count, latency, error rate, token consumption), backend-level (health, concurrency, queue depth), business-level (cost estimation, cache hit rate).

Admin API

• Backend management (add/update/delete/enable), pool management (create/configure/status), ops (failover, scale, log view).

OpenAI Compatibility

• Zero-migration for existing OpenAI SDK apps, supports chat/completions, embeddings, models endpoints, and features like function calling/streaming.

1. Unified Gateway: Single entry for all LLM requests (ideal for enterprise resource sharing, access control, cost optimization).
2. Multi-Tenant Isolation: Independent pools per tenant (for SaaS providers, data isolation needs).
3. Edge-Cloud Hybrid: Edge nodes handle low-latency requests, cloud handles complex tasks (IoT, mobile apps).
4. A/B Testing: Traffic splitting for model comparison (evaluate new model effects).

Performance Optimizations

• Connection pooling (HTTP/2 multiplexing), request batch processing, response caching (hash-based with TTL), streaming optimization (SSE, backpressure control).

Ops Integration

• Prometheus+Grafana (real-time dashboards, alerts), structured logging (ELK/Loki compatible), OpenTelemetry tracing (end-to-end link analysis).

Security Measures

• Auth: API Key management, RBAC, request signing.
• Data Protection: TLS encryption, sensitive info desensitization, audit logs.
• Rate Limiting: Global, tenant-level, adaptive.

Comparison with Alternatives

Feature	llm-pool	LiteLLM	BentoML
Multi-backend Support	Yes	Yes	Yes
OpenAI Compatibility	Yes	Yes	Partial
Scheduling Policies	Rich	Basic	Basic
Replica Management	Native	No	K8s-dependent
Metrics	Built-in	External	External
Admin API	Full	Basic	Basic
Complexity	Medium	Low	High

Summary llm-pool is a production-ready LLM pooling solution that unifies multi-backend management, intelligent scheduling, and observability. Its FastAPI base ensures performance, while OpenAI compatibility reduces migration costs.

Future Directions

• Reinforcement learning-based scheduling.
• Auto model quantization selection.
• Federated learning support.
• Fine-grained cost allocation.

It's a valuable middleware for teams building LLM infrastructure, suitable for small unified gateways to large multi-tenant platforms.