AI Gateway: AWS Cloud-Native Practice for Enterprise-Grade LLM Inference Gateway

AI Gateway is an enterprise-grade LLM inference gateway solution built on AWS cloud-native architecture. It addresses key challenges in enterprise LLM applications: unified access to multiple model providers (Bedrock, OpenAI, Anthropic, Google, Azure OpenAI), security assurance, cost control, and observability. Core features include Cognito M2M authentication, ALB native JWT validation, ECS Fargate containerization, CloudWatch observability, comprehensive security scanning, and supply chain protection. It is based on Portkey AI Gateway OSS and designed for production environments.

With the widespread application of large language models in enterprises, technical teams face critical challenges: how to unify API access to multiple model providers, ensure security, and control costs. AI Gateway was designed to solve these issues by providing a lightweight, production-ready LLM access layer that supports OpenAI Chat Completions and Anthropic Messages formats, along with auto-scaling and cloud-native best practices.

The infrastructure uses a single-region, dual-availability zone deployment. Key components:

• Network layer: VPC with 2 public subnets (ALB) and 2 private subnets (ECS tasks), NAT gateway for outbound access, VPC endpoints for AWS services (ECR, CloudWatch Logs, Secrets Manager, S3).
• ALB: TLS 1.3 encryption, WAF v2 (AWS managed rules + IP rate limits), native JWT validation (avoids API Gateway cost).
• Authentication: Cognito user pool for M2M (client_credentials grant, custom OAuth scopes, JWKS for ALB signature verification).
• Compute: ECS Fargate running Portkey gateway + AWS OpenTelemetry Collector sidecar, with auto-scaling based on CPU and ALB requests.

Comprehensive security covers development to production:

• SAST: Semgrep (OWASP Top10), Bandit (Python-specific), CodeQL (GitHub semantic analysis).
• Secret detection: Gitleaks (pre-commit hooks).
• IaC scanning: Checkov (2500+ policies) and TFLint for Terraform.
• Container security: Hadolint (Dockerfile best practices), Trivy (vulnerability scans), Syft (SBOM generation), Cosign (image signing).
• Supply chain: Excluded LiteLLM due to 14 known CVEs (including RCE, SSRF) and later validated by its 2026 supply chain attack (documented in ADR).

The flow ensures zero extra cost and latency:

1. Client requests JWT from Cognito oauth2/token (client_credentials, client ID/secret).
2. Cognito returns signed JWT (1-hour validity, scope claims).
3. Client sends JWT in Authorization header to ALB.
4. ALB validates JWT via Cognito JWKS (checks issuer, expiry, scope).
5. Valid requests are forwarded to ECS Fargate; invalid ones get 401 (no backend forwarding). This offloads auth to ALB, avoiding API Gateway or Lambda overhead.

Observability: CloudWatch logs (gateway + OTel collector), pre-defined Logs Insights queries, dashboards (request volume, error rate, latency, provider stats). OTel collector sidecar sends traces to X-Ray, metrics (EMF) and logs to CloudWatch. Dev experience: mise (tool version manager for Python, Terraform, etc.), mise.toml for task management (install, test, scan). Lefthook git hooks (pre-commit: ruff, pyright, Gitleaks, Hadolint; pre-push: full tests/scans). Conventional Commits enforced.

Applicable Scenarios: Organizations needing multi-provider switching, compliance-focused enterprises, teams reducing API Gateway costs, platform teams managing LLM traffic. ADR docs detail key decisions (e.g., Portkey over LiteLLM, ALB JWT over API Gateway). Summary: AI Gateway integrates security, observability, cost control, and dev experience, serving as a mature cloud-native reference for enterprise LLM applications.