Vetch: An Observability Tool for Energy Consumption and Cost of LLM Inference

Vetch is an energy consumption and cost observability tool launched by Prismatic Labs, designed specifically for large language model (LLM) inference scenarios. It aims to address the pain point where energy consumption and cost during the LLM inference phase are often overlooked. It helps developers and enterprises track the energy consumption and financial costs of LLM calls in real time, supports model selection decisions, cost control, and green AI practices, filling an important gap in the AI infrastructure field.

With the widespread application of LLMs across various industries, the issues of energy consumption and cost during the inference phase have gradually become prominent. Although the high energy consumption of LLM training is well-known, the energy consumption of the inference phase (e.g., hundreds of millions of daily queries for ChatGPT) may be several times that of training, yet it is often overlooked. The Vetch project targets this pain point and provides a professional observability solution for energy consumption and cost.

Environmental Sustainability Considerations

The high energy consumption of LLM training is well-known, but the energy consumption during the inference phase is also considerable (e.g., the daily query energy consumption of ChatGPT may be several times that of training). Under the goal of carbon neutrality, it is the responsibility of technical practitioners to understand and optimize the energy footprint of AI.

Cost Control Needs

LLM API call costs have become an important operational cost for enterprises. The existing token-based billing method is difficult to reflect the actual cost structure, requiring fine-grained analysis to support budget planning and resource optimization.

Decision Support for Model Selection

Different LLMs have trade-offs between performance, cost, and energy consumption. Vetch's data can help developers make more informed decisions when selecting models.

Real-Time Energy Consumption Tracking

• Energy consumption estimation for single requests
• Cumulative energy consumption statistics
• Energy consumption comparison across different models
• Energy consumption trend analysis over time

Cost Analysis and Prediction

• API cost allocation by project/application
• Cost trend prediction and budget alerts
• Price comparison across different providers
• Cost optimization recommendations

Observability Integration

Supports seamless integration with mainstream platforms such as Prometheus and Grafana, enabling unified visual display and alert management.

Enterprise-Level LLM Application Management

• Identify high-cost API call patterns
• Optimize prompts to reduce token consumption
• Implement quota management and usage policies
• Generate compliance reports to meet ESG requirements

Green AI Practices

• Quantify and visualize LLM carbon footprints
• Develop carbon neutrality roadmaps
• Demonstrate environmental commitments to stakeholders
• Optimize model selection to reduce environmental impact

Developer Efficiency Tool

• Avoid unexpected high API bills
• Cultivate efficient prompt writing habits
• Consider cost-effectiveness in the prototype phase

Complexity of Energy Consumption Estimation

LLM inference energy consumption is affected by multiple factors such as model size, input/output length, hardware configuration, and batch processing strategy. Vetch needs to establish a reliable model to convert API calls into energy consumption estimates.

Cross-Provider Data Integration

Different LLM providers have different API response formats and billing methods. Vetch needs to abstract a unified monitoring interface to support unified observation across multiple clouds and models.

Balance Between Real-Time Performance and Accuracy

It is necessary to find a balance between real-time monitoring and estimation accuracy, avoiding performance overhead due to excessive precision or loss of practical value due to rough estimation.

From Performance-First to Efficiency-First

After LLM applications enter the production environment, efficiency indicators such as energy consumption, latency, and cost have become increasingly important. Vetch is a product of this trend.

Expansion of Observability Boundaries

Traditional application observability focuses on latency, error rates, etc. Vetch extends this to energy and cost dimensions, representing an innovative direction in the observability field.

Responsible AI Practices

Energy consumption monitoring is an important part of responsible AI. Vetch helps developers and enterprises make more responsible AI decisions through transparent data.

Vetch fills the gap in the observability of energy consumption and cost for LLM inference in the AI infrastructure field, representing an AI development concept that emphasizes intelligence alongside efficiency, cost, and sustainability. As AI regulation improves and ESG requirements increase, such tools will become more important. In the future, we can expect more energy-efficient model architectures, intelligent inference scheduling, and improved carbon footprint tracking systems. It is recommended that teams using LLMs in production environments establish energy consumption and cost observability as early as possible to ensure the long-term sustainable development of their projects.