Synapse: A Unified AI Inference Gateway Integrating LLM, Speech, and Audio Processing

Synapse is a unified AI gateway built on FastAPI, designed to provide a single entry point for all AI workloads in a K3s cluster and address the pain points of integrating multiple AI services. It supports various services including LLM embedding, chat completion, TTS, STT, speaker separation and verification, and audio noise reduction, and offers OpenAI-compatible APIs to simplify development and operation processes.

In current AI application development, developers need to interface with multiple backend services (LLM, TTS, STT, etc.), each with different API formats, authentication methods, and deployment requirements, which increases development complexity and operational difficulty. Synapse was created to address this pain point.

Synapse adopts a two-layer communication model: gRPC over TCP is used between clients and the gateway, and between the gateway and backend replicas to support streaming transmission. The cluster integrates six major backend services (llama-embed, llama-router, Chatterbox TTS, whisper-stt, pyannote-speaker, deepfilter-audio), and the gateway intelligently routes requests to the corresponding service based on request type.

Synapse exposes 22 OpenAPI endpoints, including OpenAI-compatible /v1/embeddings and /v1/chat/completions; it provides a web dashboard (e.g., /ui) to view health status, operate models, and access real-time logs; it supports voice library management (uploading samples, cloning voices), with data persisted via PVC.

Synapse configures circuit breakers and retry mechanisms for backends, enabling automatic failover in case of failures; the aggregated health check endpoint /health returns the status of all backends; it supports Redis as a terminal log sharing bus, allowing unified visibility of log streams under multiple replicas.

A Makefile is provided to simplify deployment (from infrastructure to backend services); it supports Forge for remote image building; configurations are mounted as ConfigMaps via YAML files, and environment variables allow customization of log levels, Redis parameters, etc.

It is suitable for teams that manage multiple AI services uniformly in a K3s cluster, projects that need OpenAI-compatible APIs but use open-source models, and conversational AI applications that integrate speech processing. Developers can call multiple AI capabilities through a unified interface without worrying about underlying details.

Synapse represents the evolution direction of AI infrastructure—encapsulating heterogeneous AI capabilities through the gateway layer to provide consistent and reliable interfaces. As AI services increase, the unified gateway model will become more important, providing a reference solution for AI platform construction.