InferGuard: A Read-Only Diagnostic and Observability Tool for Large Model Inference Services

InferGuard is a diagnostic tool designed specifically for distributed large model inference services. It supports mainstream inference engines such as vLLM, SGLang, Dynamo, and llm-d, providing read-only observability and troubleshooting capabilities to address complex challenges in distributed inference operation and maintenance.

As large models scale up, distributed inference has become mainstream, but it brings complex operation and maintenance issues: it is difficult to quickly locate problems in services with tens to hundreds of GPU nodes; traditional monitoring tools have coarse-grained metrics and cannot penetrate into the engine's internal operations; diagnostic tools that are improperly designed can easily interfere with services, making read-only diagnostic methods crucial.

InferGuard is an open-source project developed by Touchdown Labs, focusing on read-only diagnostic capabilities for distributed large model inference services. Its design philosophy emphasizes a balance between security and observability: all operations are read-only and do not modify system states; it supports integration with existing monitoring stacks such as Prometheus and Grafana, outputting standardized metrics or detailed reports.

InferGuard supports four mainstream engines:

• vLLM: Monitors KV cache utilization, scheduling queue length, batch processing efficiency, etc.;
• SGLang: Tracks syntax constraint compilation status, generation latency distribution, and correctness of structured outputs;
• Dynamo: Analyzes batch processing strategy effectiveness, request priority scheduling, and GPU resource utilization;
• llm-d: Provides visibility into plugin systems, backend switching status, and cross-backend performance comparisons.

Core features include:

1. Multi-level metric collection: System layer (GPU utilization, memory, PCIe bandwidth), engine layer (request queue depth, batch size, cache hit rate), application layer (end-to-end latency, token generation rate, first token time);
2. Correlation analysis: Correlates metrics across nodes/engines to help identify the root cause of performance bottlenecks (e.g., analyzing resource competition and network conditions when node latency is abnormal).

Best practices for using InferGuard:

• Perform in-depth diagnosis during off-peak hours to avoid affecting service performance;
• Configure appropriate permissions to access only necessary metric endpoints;
• Store diagnostic data separately from logs to avoid becoming an attack target;
• Regularly review access logs to ensure that only authorized personnel view sensitive information.

InferGuard fills the gap in secure non-intrusive diagnosis in the distributed large model inference ecosystem, meeting the operation and maintenance needs of enterprise-level production inference services. It promotes the operation and maintenance concept of "observability built into inference services", helping to build reliable and maintainable AI infrastructure to support large-scale LLM deployments.