Tracechat: A Multi-Provider LLM Observability Workspace for Production Environments

Tracechat is an open-source full-stack AI chat application designed to showcase best practices for observability in multi-provider LLM applications for production environments. It covers core capabilities such as multi-turn conversation management, multi-LLM provider integration, streaming responses, event-driven log ingestion, PII redaction, and visual dashboards, providing developers with a complete reference for observability architecture.

As LLMs move from experimentation to production, traditional monitoring solutions struggle to handle their probabilistic output characteristics, failing to effectively capture key information such as token usage, response latency, model distribution, and PII leakage risks. As an educational reference project, Tracechat fills this gap by demonstrating a complete observability pipeline.

Tracechat supports multi-turn conversation context management, allowing creation/restoration of conversation threads. It has built-in support for multiple providers like OpenAI, Google Gemini, and Groq, with runtime model switching (falling back to simulation mode if no API key is configured). It uses Server-Sent Events (SSE) to implement streaming responses, displaying model-generated content in real time to enhance user experience.

Tracechat's observability system uses a layered architecture:

1. Instrumented LLM Wrapper: Non-intrusively collects metadata (provider/model, latency, token usage, PII-redacted content, etc.);
2. Ingestion Endpoint and Queue: Validates payloads via /api/ingest/inference, records raw events, and publishes them to Redis/BullMQ queues;
3. Ingestion Worker: Consumes queue events and writes to PostgreSQL;
4. Data Model: Includes four core entities—Conversation, ChatMessage, InferenceLog, and IngestionEvent—with indexes optimized for queries.

Tracechat implements regex-based PII redaction, automatically identifying and masking sensitive information such as emails, phone numbers, and API keys to ensure sensitive data does not enter persistent storage. This solution serves as a practical baseline, not a full data loss prevention system.

Dashboard displays key metrics: throughput, latency (average/quantiles), token usage distribution, error rate, and number of canceled requests. Deployment Methods:

• Local Development: Start via .env configuration, npm installation, and migrations;
• Docker Compose: One-click startup of all services (PostgreSQL, Redis, API, worker, frontend);
• Kubernetes: Provides complete self-hosted manifests covering all components and configurations.

Current Limitations: Lack of authentication, no retry mechanism for ingestion, missing Anthropic/local model adapters, insufficient dashboard functionality, and low automated test coverage. Improvement Directions: Add user authentication and authorization, implement ingestion retries, support more providers, enhance the dashboard, and supplement automated tests.

Tracechat provides LLM application developers with a blueprint for observability architecture. Key insights include:

1. Observability should be built-in rather than bolted on;
2. Use message queues to decouple critical paths from auxiliary functions;
3. Privacy protection (PII redaction) should be a default behavior;
4. Design should account for the heterogeneity of multiple LLM providers. It has significant reference value for teams transitioning from prototypes to production.