Tetrics: A Continuous Evaluation Framework for LLM-Driven Development Tools

Tetrics is a domain-agnostic continuous evaluation framework prototype designed specifically for LLM-driven development tools. Based on a 20-month longitudinal study and using the Goal-Question-Metric (GQM) methodology, it aims to help enterprises systematically evaluate and monitor the quality and stability of AI programming tools, addressing the issue that the traditional "one-time evaluation" model cannot adapt to the dynamic iterative nature of LLM tools.

With the popularity of AI programming assistants like GitHub Copilot and Claude Code in development workflows, the continuous iteration of LLM tools (model version updates, prompt optimization, architecture adjustments) has rendered the traditional one-time evaluation model obsolete. Enterprises face a core dilemma: how to make informed technical decisions in an unstable ecosystem? For example, a model that performed excellently last month may degrade later, and architecture changes in third-party services can affect downstream tools.

Tetrics was developed by developers such as Eneko Pizarro and is the implementation product of the paper Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation. Based on a longitudinal study spanning 6 cycles from March 2024 to October 2025, key findings were revealed:

• Quality Volatility: High-scoring models may degrade later
• Hidden Dependency Risk: GitHub Copilot architecture changes affect integrated models
• Availability Crisis: Some high-performance models suddenly become unavailable
• Customization Advantage: Custom agents are 20-90% higher in quality than general-purpose tools
• Necessity of Continuous Monitoring: Single evaluations cannot detect time-dimensional quality patterns

Tetrics adapts the GQM methodology (Goal → Question → Metric), and its core components include:

• Metric Engine: Automated metrics (compilation success rate, test coverage, etc.) + expert manual evaluation for double verification
• Evaluation Cycle Management: Multi-model tracking (GPT-4, Claude, etc.) + configuration change records to identify long-term trends
• API-First Architecture: RESTful services built with FastAPI for easy CI/CD integration
• Persistent Storage Layer: PostgreSQL + Alembic to ensure data traceability and schema evolution

Tetrics uses a modern tech stack:

• Backend: Python3.11 + Poetry + FastAPI
• Frontend: Next.js evaluation dashboard
• Authentication: Keycloak enterprise-level identity verification
• Deployment: Docker Compose orchestration Project structure: app (FastAPI application), alembic (database migration), front (frontend), keycloak-config (authentication configuration), docker-compose.yml (service orchestration)

Tetrics is applicable to various industrial scenarios:

• Technical Selection Decision: Establish benchmark tests to compare candidate tools with existing solutions
• Vendor Risk Management: Monitor LLM service providers to detect model degradation or service decline in a timely manner
• Prompt Engineering Optimization: Quantitatively evaluate the effectiveness of different prompt strategies
• Compliance and Audit: Provide auditable records to support decisions in regulated industries

Tetrics fills a key gap in the field of LLM tool evaluation, helping enterprises shift from blind following to data-driven AI tool adoption strategies. In the future, as LLM penetration increases, more evaluation frameworks targeting specific domains (such as safety-critical systems and financial software) are expected to emerge, forming a complete quality assurance ecosystem.