Strike: Real-Time Cost and GPU Monitoring Tool for Self-Hosted LLM Inference

Strike is a lightweight Go-language Sidecar proxy designed specifically for self-hosted large language model (LLM) inference services. It provides real-time cost calculation and GPU usage monitoring capabilities, helping teams accurately track resource consumption and cost overhead for each inference request, and addressing the pain points of cost tracking in self-hosted scenarios.

Unlike cloud-hosted APIs, self-hosted LLM inference requires teams to manage infrastructure on their own. Cost calculation involves multiple dimensions such as GPU rental/depreciation, electricity, and bandwidth. Resource consumption varies greatly across different requests, and the lack of fine-grained visibility makes it difficult to optimize resource allocation and cost sharing. Traditional monitoring tools only focus on system metrics and lack the business context of LLM inference (e.g., "cost of a specific request").

Strike is deployed in Sidecar mode, running as an independent process alongside the inference service. It requires no modification to existing code and supports zero-intrusion integration with multiple frameworks such as vLLM and TensorRT-LLM. Written in Go, it has low resource usage and efficient concurrent processing. It follows cloud-native principles, supports containerized deployment, seamlessly integrates with orchestration tools like Kubernetes, and offers flexible configuration.

Core capabilities include cost calculation and resource monitoring: Cost calculation supports custom models (e.g., GPU hourly rates, tiered pricing) and outputs the cost of each request in real time. Resource monitoring tracks LLM-specific metrics such as GPU utilization, token generation rate, and peak memory usage. Metrics are exposed via Prometheus export, REST API, and platforms like Datadog/New Relic.

Deployment is simple: In containerized environments, you can add a Strike container and configure network rules; on bare metal, it can run as a system service. Integration supports defining cost models via YAML/environment variables, and cost sharing by tags/namespaces in multi-tenant scenarios. It provides request tracing functionality to track resource consumption throughout the entire lifecycle of a request via a unique ID.

It is suitable for scenarios such as multi-team shared inference clusters (cost sharing), cost optimization and capacity planning (identifying inefficient loads), performance tuning (analyzing model bottlenecks), and anomaly detection (identifying resource exhaustion or abusive requests), helping teams achieve refined cost management and resource optimization.

Strike fills the gap of general monitoring tools in self-hosted LLM scenarios, providing teams with refined visibility into inference costs, and is an important infrastructure component for LLMOps practices. As the demand for self-hosted LLMs grows, such specialized tools will become increasingly important and are worth considering for technical selection by relevant teams.