Argus: A Chatbot Framework for LLM Inference Observability with Native OpenTelemetry Support

Argus is an open-source LLM chatbot framework based on TypeScript. Its core innovation lies in natively integrating OpenTelemetry (OTel) observability capabilities into the LLM inference process, enabling real-time WebSocket streaming and full distributed tracing. It addresses the pain point of traditional LLM applications needing to additionally integrate monitoring SDKs, providing developers with a solution that combines conversational interaction and in-depth model monitoring.

In modern AI application development, observability is as important as functionality. Traditional LLM applications often need to integrate monitoring SDKs outside of business code, but Argus treats observability as a first-class citizen, ensuring every model call can be fully traced and analyzed at the architectural level. It aims to provide a complete solution that combines conversational interaction and in-depth model behavior monitoring.

Real-time WebSocket Streaming

When users interact with the bot, each token generated by the model is pushed to the client instantly via WebSocket, enhancing responsiveness and front-end interaction possibilities.

Native OpenTelemetry Integration

Each LLM inference call automatically generates tracing data compliant with OTel specifications, including request context, input/output records, latency and token consumption statistics, error status, etc. It can be integrated with Jaeger, Zipkin, or cloud APM platforms for full-link monitoring.

Modern Tech Stack

It uses TypeScript, along with a Monorepo architecture (pnpm workspace), Turbo build, end-to-end testing, and infrastructure as code (infra directory).

1. Production Environment Monitoring: An out-of-the-box observability solution that can be integrated into existing monitoring systems via the OTel protocol without the need for manual instrumentation.
2. Model Behavior Analysis: Identify performance bottlenecks or anomalies through tracing data, providing support for model optimization and prompt engineering.
3. Debugging and Troubleshooting: Distributed tracing capabilities quickly locate the root cause of issues (prompt errors, model API anomalies, network timeouts, etc.).

Argus uses a clearly layered directory structure:

• apps/: Main chatbot service
• packages/: Shared libraries and core modules
• infra/: Infrastructure configurations (Docker, K8s, or Terraform definitions)
• docs/: Project documentation
• tests/e2e/: End-to-end test cases This structure facilitates team collaboration and long-term evolution.

Argus is open-sourced under the MIT license, with active Pull Requests on GitHub indicating community interest. It represents a trend in AI application development: complete functionality while emphasizing observability and engineering practices.

Argus demonstrates the combination of LLM application development and cloud-native observability best practices. It is a usable framework for production-grade AI application developers and an excellent reference for OTel-integrated AI workflows. It is recommended that developers prioritize observability when building LLM applications to ensure service stability and reliability.