LLM Observatory: An Observability Platform for Large Language Models

This article introduces the open-source project LLM Observatory, an observability platform for large language models. Starting with a lightweight Go API connecting to Ollama, the project gradually evolves into a complete AI application observability stack, providing three-in-one monitoring capabilities of Metrics, Logs, and Tracing. It aims to address the operation and maintenance (O&M) and monitoring needs of LLM workloads in production environments. The project is maintained by ltcwr, with source code hosted on GitHub (link: https://github.com/ltcwr/llm-observatory), and was released on June 4, 2026.

Most AI projects focus on building the applications themselves, while LLM Observatory focuses on the understanding, monitoring, and O&M of LLM workloads. It fills an important gap in AI infrastructure: as more LLMs are deployed in production environments, O&M personnel need to understand model response latency, performance-cost comparisons between different models, error request patterns, token consumption trends, etc. However, general observability tools struggle to cover LLM-specific metrics (such as token count, generation latency, prompt complexity). This project aims to provide complete visibility for large language models in production environments.

Current Phase: Provides a Go API based on the Gin framework, responsible for forwarding requests to the local Ollama instance. The data flow is Client → Gin API → Ollama → Model (e.g., Qwen). Evolution Roadmap:

• Phase 1 (Metrics): Integrate Prometheus to provide request counters, latency metrics, error tracking, token generation metrics, paired with Grafana dashboards (including model comparisons) and performance analysis.
• Phase 2 (Logs): Integrate Loki to implement centralized log collection and request trace identifiers.
• Phase 3 (Tracing): Support OpenTelemetry, integrate Tempo to achieve end-to-end request tracing.

Long-term Deployment Vision: Client → API Gateway → LLM Observatory → Ollama/vLLM → Models. Observability Data Flow: Metrics → Prometheus; Logs → Loki; Traces → Tempo, with unified display in Grafana eventually. O&M Feature Planning: Docker support, Kubernetes deployment, Helm Charts, horizontal scaling, multi-model support, cost estimation, token analysis, model health monitoring, AI workload observability dashboard.

The core technology stack used in the project includes:

• Language: Go 1.22+
• Web Framework: Gin
• Inference Engine: Ollama
• Monitoring Tools: Prometheus + Grafana
• Logging Tools: Loki
• Tracing Tools: OpenTelemetry + Tempo
• Containerization: Docker + Kubernetes

Steps to run LLM Observatory:

1. Start Ollama and run a model: ollama run 'your-model'
2. Start the Observatory service: go run .
3. The service will run at http://localhost:8080. You can send requests via the POST /chat interface (e.g., {"prompt": "What is Kubernetes?"}) to get responses.

Compared to commercial LLM observability platforms like LangSmith and Langfuse, LLM Observatory has the following features:

• Fully open-source, allowing users to control their data;
• Deep integration with the open-source ecosystem (Ollama, Prometheus, Grafana, etc.), lowering the adoption threshold;
• Provides an observability evolution path from development to production, without the need for initial investment in complex commercial solutions.