Goose Token Tracker: A Token Usage and Cost Tracking Proxy Built for Local LLM Inference

This article introduces the open-source tool Goose Token Tracker, which aims to solve the problems of usage monitoring and cost control in local large language model (LLM) deployment. Using reverse proxy technology, this tool enables token usage tracking, cost calculation, and vLLM performance metric collection, helping enterprises address the hidden cost blind spots and resource allocation challenges of local deployment, and optimize AI return on investment.

Enterprises often overlook cost monitoring when shifting to local LLM deployment. The hidden costs include hardware depreciation, power consumption, operation and maintenance labor, and opportunity costs. Without usage data, it is difficult to evaluate the economic viability of local deployment. Additionally, in scenarios where multiple teams share model services, the lack of usage tracking leads to unfair resource allocation—for example, overuse crowds out other teams' resources or low-priority tasks occupy critical business computing power.

Goose Token Tracker adopts a reverse proxy architecture, positioned between the client and the LLM inference service. It has the advantages of zero intrusion (no need to modify client code) and protocol compatibility (supports OpenAI API, vLLM native interface, etc.). For token metering, it has built-in support for mainstream tokenizers (tiktoken, SentencePiece, etc.), and achieves real-time accurate counting of streaming responses through incremental parsing technology. It is also deeply integrated with vLLM, collecting performance metrics such as request latency distribution, first token time, throughput, GPU utilization, and KV cache hit rate, which helps with capacity planning and optimization.

This tool supports cost calculation and multi-dimensional allocation: After users configure parameters for hardware, power, and operation and maintenance costs, the system automatically calculates the allocated cost for each call, and can generate reports based on project, team, application, or user dimensions, facilitating internal settlement and ROI analysis. In addition, it provides real-time monitoring dashboards and anomaly detection functions, allowing custom views and setting budget/performance threshold alerts; it supports data export (CSV, JSON, Parquet) and integration with monitoring systems such as Prometheus and Grafana, and API interfaces allow external systems to query for automated cost control and resource scheduling.

1. A technology company used Goose Token Tracker to monitor its internal code assistant service, found that night batch processing tasks occupied a lot of resources, and reduced costs by 30% after adjusting the scheduling strategy; 2. Another enterprise used the cost allocation function to charge AI resource usage fees to various business departments, promoting an overall improvement in usage efficiency.

Future versions will introduce machine learning models to predict usage trends, support more granular cost attribution, and plan to integrate with model performance optimization tools. Conclusion: Goose Token Tracker fills the gap in the local LLM deployment ecosystem. By providing precise monitoring and cost calculation, it helps enterprises optimize AI investments. As local deployment becomes more popular, it will become a standard component of enterprise AI infrastructure.