LLM Observability Platform: End-to-End Monitoring Solution for AI Systems in Production

This article introduces the llm-observability project maintained by pankaj45 on GitHub, a full-stack AI observability platform for production environments. It addresses the limitations of traditional monitoring tools in LLM applications and provides complete monitoring, logging, and analysis capabilities through event-driven architecture, PII redaction, context orchestration, and real-time analytics dashboards.

As LLM applications move from prototype to production, traditional software monitoring tools struggle to address three core challenges:

1. Observability Blind Spots: Internal states of model inference are hard to track, lacking token-level latency and error rate metrics;
2. Data Privacy Risks: Directly recording user inputs containing PII leads to compliance issues;
3. Complex Context Management: Continuity of multi-turn conversation states is difficult to capture, making traditional stateless API monitoring ineffective.

llm-observability uses a microservices architecture, event-driven communication, and layered data storage:

• Core Services: Inference Gateway (handles requests, PII redaction, context orchestration), Ingestion Worker (consumes Kafka events and writes to ClickHouse), Analytics Query Service (provides queries for dashboards), Next.js Frontend (chat UI and analytics dashboards);
• Layered Storage: PostgreSQL (OLTP layer for core entities), ClickHouse (analytics layer for efficient aggregation), Redis (coordination layer for short-term state caching).

PII Redaction: Uses the PiiRedactionPort interface to scan messages with regex, replacing PII with placeholders (e.g., [EMAIL]). Redaction occurs before persistence, so the model never receives raw PII. It supports 6 types of PII (email, phone, credit card, etc.); Context Orchestration: Automatically injects runtime context (date, time), ToolNeedRouter triggers backend tools (e.g., CoinGecko, Tavily Search), and PostgreSQL manages conversation states to ensure integrity.

SSE Streaming Architecture: Uses Server-Sent Events for real-time push, defining multiple event types (request.accepted, token.delta, tool status, etc.) to display inference progress; Analytics Dashboards: Built on Grafana to show key metrics: latency (P50/P95/P99), throughput (requests per second, token rate), error rate, cost estimation, with data from ClickHouse.

Deployment: Local deployment uses Docker Compose for one-click startup (Makefile encapsulates commands like dev-ready/dev), and production supports Kubernetes deployment; Logging Strategy: Does not record raw content, only metadata (hash, token count, latency, etc.). Tool calls only log metadata, with structured JSON logs and support for OpenTelemetry tracing.

Core insights from the project:

1. Observability requires end-to-end design;
2. Privacy protection is an architecture-level feature;
3. Event-driven approach achieves service decoupling;
4. Layered storage optimizes costs. For LLM application teams, it is both a deployable solution and an architectural design reference document, worth studying and learning from.