Ollive: A Practical Guide to Building Production-Grade LLM Inference Observability Systems

This article analyzes the open-source project Ollive, which aims to build a complete inference observability system for LLM applications. It addresses the challenges of LLM inference monitoring in production environments through core features like SDK encapsulation, asynchronous log collection, PII desensitization, and visual dashboards. The original author is 524himanshu, the project is open-sourced on GitHub, and it was released on May 25, 2026.

With the widespread application of LLMs in production, traditional API monitoring methods are insufficient—LLM calls have non-deterministic, high-latency, and high-cost characteristics, requiring monitoring of token consumption, response latency, privacy exposure risks, and output quality. The Ollive project was thus born, offering a complete solution including a lightweight SDK, data ingestion pipeline, PII desensitization mechanism, and visual dashboard.

Ollive uses a three-tier architecture: frontend with Next.js + Tailwind CSS; backend with FastAPI + SQLAlchemy; data layer supporting PostgreSQL (production) and SQLite (development). It can be deployed with one click via Docker Compose. The core innovation is the SDK layer (e.g., GeminiSDK) that captures telemetry data non-intrusively, and asynchronous log transmission (fire-and-forget mode) ensures no impact on core function latency.

The SDK automatically captures key metrics for each inference: model name/provider, start/end time, latency, token count, call status, etc., and also records a 200-character preview of input and output. Log sending uses try-except fault tolerance to prevent observation infrastructure failures from affecting the product. Accurate token counting is based on usage metadata returned by Gemini, facilitating cost analysis and optimization.

Ollive has a built-in PII desensitization mechanism that detects and replaces sensitive information such as emails, phone numbers, and SSNs via regular expressions. The server sets a pii_detected field to prevent tampering. While the current regex solution is lightweight, its accuracy is limited in complex scenarios; the documentation recommends using professional NLP desensitization tools like Microsoft Presidio in production environments.

The database uses three core tables: conversations for conversation metadata, messages for message content, and inference_logs for telemetry data, separating UX and operation data. UUIDs are used as primary keys (to avoid revealing quantity and facilitate distribution), and preview fields (200 characters) balance storage efficiency and debugging needs.

Deployment optimizations: Three-step startup with Docker Compose (copy configuration, add Gemini key, docker compose up); local development supports Docker-free solutions (Python virtual environment + Node.js server); FastAPI automatically generates Swagger UI for easy interface testing.

Ollive provides an excellent reference for LLM observability, with core principles including non-intrusive collection, asynchronous transmission, and defensive programming. Future improvement directions include support for streaming responses, event-driven architecture, time-series data visualization, multi-model provider support, user authentication, K8s deployment, etc.