Google EvalBench: A Generative AI Evaluation Framework for Database Tasks, Supporting NL2SQL and Multi-Database Dialect Evaluation

Google Cloud Platform's open-source EvalBench is a modular evaluation framework designed specifically for assessing the performance of generative AI on database tasks (especially NL2SQL). It supports the evaluation of three SQL types: DQL, DML, and DDL, and has A/B testing and detailed result analysis capabilities. It addresses core challenges in NL2SQL evaluation such as execution validation, multi-dialect adaptation, and fine-grained quality assessment, providing an end-to-end evaluation loop.

NL2SQL is an important enterprise application scenario for large language models, but its evaluation faces three major challenges: 1. Execution correctness validation (requiring an actual database environment to verify syntax and execution results); 2. Multi-database dialect support (handling differences between MySQL, PostgreSQL, BigQuery, etc.); 3. Fine-grained quality assessment (efficiency, edge cases, optimality, etc.). EvalBench builds a complete evaluation pipeline to address these issues.

EvalBench supports evaluation of three types of SQL tasks:

• DQL: Verify the semantic correctness and result consistency of SELECT queries;
• DML: Safely manage test environments for modification operations like INSERT/UPDATE/DELETE;
• DDL: Evaluate the ability to understand database schemas for structural operations like CREATE/ALTER/DROP. It fully covers the complete database workflow, not just simple queries.

The framework adopts a modular design, with the core being a customizable evaluation pipeline (input → generate SQL → execute → score). Key modules include:

• Extensible scoring strategies (built-in/custom logic);
• Data processor (parse datasets, manage test environments);
• Result storage (local CSV/BigQuery) and dashboard visualization; It supports "plug-and-play" extensions to adapt to different needs.

When the result storage is configured as BigQuery, it supports:

• Experiment creation: Parallel comparison of different model configurations and prompt strategies;
• Performance quantification: Fine-grained analysis of the impact of improvements on specific dialects/query types;
• Regression analysis: Highlight query-level changes, provide LLM-assisted score explanations, and distinguish between improvements and regressions. It becomes a work platform for NL2SQL model development and optimization.

EvalBench is suitable for multiple types of users:

• Model developers: Standardized evaluation to identify model strengths and weaknesses;
• Prompt engineers: A/B testing to quantify strategy effectiveness;
• Enterprise users: Benchmarking NL2SQL solutions;
• Researchers: Building new NL2SQL benchmarks.

EvalBench lowers the threshold for evaluating NL2SQL systems, providing flexible, scalable, production-ready infrastructure. Whether for automated testing, model selection, or academic research, it can provide objective support to promote the development and application of NL2SQL technology.