Squirrel LLM Gateway: Enterprise-Grade Unified Access Gateway for Large Language Models

This article introduces Squirrel LLM Gateway, an open-source enterprise-grade LLM proxy service. It supports unified access to multiple providers such as OpenAI and Anthropic, and features core capabilities like intelligent routing, failover, cost analysis, and a modern management panel. It aims to solve the fragmented management problem of enterprise access to multiple LLM providers and provide solid infrastructure support for enterprise-level LLM applications.

With the development of LLM technology, enterprises need to connect to multiple providers like OpenAI and Anthropic. However, different API formats, authentication methods, and pricing strategies lead to fragmented management challenges: developers have to maintain independent integration code, operations teams struggle with unified monitoring and cost control, manual switching during failures affects business continuity, and there is a lack of centralized access control and auditing capabilities.

Squirrel's core features include:

1. Unified Access Layer: Access multiple providers with one integration, compatible with OpenAI/Anthropic SDKs, and supports custom model mapping for transparent switching;
2. Intelligent Routing and Load Balancing: Offers multiple routing strategies such as round-robin, priority, weight, cost, and rule-based;
3. High Availability Guarantee: HTTP error retries, automatic failover between primary and backup providers, and support for long streaming responses;
4. Protocol Conversion: Supports smooth conversion between OpenAI Chat/Responses and Anthropic Messages.

Squirrel provides comprehensive observability and cost management:

• Request Tracing: Records the full lifecycle of requests, automatically calculates token consumption, measures latency, and aggregates statistics;
• Data Desensitization: Automatically processes sensitive information in logs to ensure compliance;
• Cost Analysis: Tracks costs of each model/provider, identifies high-cost patterns, and optimizes model selection.

The management panel built with Next.js + TypeScript offers:

• Provider Management: Add, test, and configure LLM connections;
• Model Mapping: Use a visual rule editor to define mappings from virtual models to actual providers;
• API Key Management: Generate, enable/disable, and delete keys;
• Log Viewing: Multi-dimensional filtering and search;
• Cost Statistics: Usage trend and cost analysis charts.

Deployment Methods:

1. Docker Compose (Recommended, PostgreSQL): Clone the repository and run the docker compose command;
2. Docker Single Container (SQLite): Single container + volume persistence;
3. Local Development: Python backend (3.12+) + Next.js frontend (18+). Application Access: Use the standard OpenAI SDK, set the base_url to the gateway address, and use the API key generated by the gateway.

Squirrel LLM Gateway helps enterprises efficiently manage multiple LLM providers. Its enterprise-grade design reduces integration complexity, optimizes cost-performance, and ensures business continuity. Application scenarios include: enterprises needing to connect to multiple providers, high-availability production environments, organizations requiring unified cost control and compliance, and development teams wanting transparent model switching.