ArcWatch: Real-Time GPU Cluster Monitoring and Cost Attribution Platform for Large Model Inference

ArcWatch is a professional monitoring, cost attribution, and alerting solution tailored for LLM inference scenarios. It addresses the monitoring and cost management challenges posed by the unique resource consumption patterns of LLM inference services, helping enterprises optimize their AI infrastructure investments. Its core features include real-time GPU cluster monitoring, fine-grained cost attribution, and intelligent alerting and anomaly detection.

LLM inference workloads differ from traditional tasks, featuring highly variable request lengths, unpredictable execution times due to autoregressive generation, and complex resource allocation patterns from model parallelism and pipeline parallelism. General-purpose cloud monitoring tools struggle to accurately reflect actual resource usage, while ArcWatch delves into the inference request granularity, tracking key metrics such as latency distribution, token throughput, and memory usage.

ArcWatch uses a distributed collection architecture, with lightweight agents deployed on each node to collect hardware (SM utilization, memory bandwidth, NVLink traffic) and software (batch size, queue depth, KV cache hit rate) metrics with low overhead. Data is aggregated into a central time-series database via streaming pipelines, supporting sub-second freshness. The front-end dashboard provides cluster health visualization, allowing drill-down into detailed metrics for individual GPUs, model instances, or requests.

ArcWatch introduces a multi-dimensional cost attribution model that tracks resource consumption and cloud costs by team, project, model version, and API key. Leveraging full-lifecycle request tracking—from entering the load balancer to completing GPU computation—it tags each request with context labels and correlates with cloud billing data to generate request-level cost reports. For shared GPU/multi-tenant scenarios, it implements a fair allocation algorithm based on actual resource usage.

ArcWatch has a built-in alerting system optimized for LLM inference, supporting static threshold alerts and time-series anomaly detection (identifying latency drift, throughput drops, error rate fluctuations). Alert rules cover the infrastructure layer (GPU failures, network partitions), service layer (model loading failures, batch timeouts), and business layer (API SLA violations). Notifications can be routed to channels like PagerDuty and Slack, with severity escalation policies.

ArcWatch represents the trend of specialization in AI infrastructure monitoring tools. As LLMs become core production components, the demand for specialized operation and maintenance tools is growing. Recommendations for enterprises: Monitoring needs to go deep into the semantic level of workloads, cost management should align with business metrics, and alert systems should be aware of the unique patterns of AI services. In the future, it needs to adapt to new hardware (TPUs, dedicated chips) and service paradigms (speculative decoding, prefix caching) to continuously provide visibility guarantees.