LexBench: A Large Language Model Evaluation System for Multilingual Environmental Law

LexBench is an LLM evaluation system designed for multilingual environmental law tasks, covering four key competency dimensions: information extraction, legal reasoning, numerical analysis, and hallucination detection. It builds its dataset based on real multilingual legal documents from three jurisdictions—Saudi Arabia, China, and Finland—and evaluates mainstream commercial LLMs such as GPT-4o and Claude. The evaluation finds that deep reasoning remains a shortcoming of models, and there are significant performance differences between models. The project is open-source, providing a standardized evaluation tool for the legal AI community.

As LLMs are increasingly applied in the legal field, general evaluation benchmarks struggle to capture the uniqueness of legal texts (complex terminology, cross-jurisdictional differences, high precision requirements). LexBench focuses on the vertical domain of environmental law and incorporates multilingual factors into a systematic evaluation framework for the first time.

LexBench simulates real legal work scenarios, with evaluation tasks covering four dimensions:

1. Information Extraction: Accurately extract key entities such as regulatory clauses and responsible parties from legal texts;
2. Legal Reasoning: Test multi-step logical deduction ability, distinguishing between text matching and true legal understanding;
3. Numerical Analysis: Evaluate the ability to understand and calculate numerical values such as fines and emission limits;
4. Hallucination Detection: Detect the model's factual accuracy and self-calibration ability in legal contexts.

The LexBench dataset collects real environmental legal documents from three jurisdictions:

• Saudi Arabia: Arabic texts, representing complex writing systems of non-Latin language families;
• China: Chinese documents, testing understanding of ideographic characters and unique legal terminology;
• Finland: Finnish texts, challenging the ability to process minor European languages. All documents are kept in their original languages without translation, testing the model's cross-language transfer and low-resource language processing capabilities.

LexBench evaluated mainstream LLMs such as GPT-4o, Claude, Gemini, and DeepSeek, with preliminary findings:

• Information extraction performed best: Basic text understanding capabilities are mature;
• Deep reasoning remains a shortcoming: Performance drops significantly in multi-level legal logical reasoning;
• Significant differences between models: Claude excels in reasoning, GPT-4o is well-balanced overall, and DeepSeek is relatively weak in hallucination control.

LexBench is implemented in Python, based on the Replit platform, and calls LLM services via standard APIs. The significance of open-source release:

1. Provide a standardized performance comparison benchmark for legal tech researchers;
2. The multilingual design serves as an important resource for cross-language legal AI research;
3. The dedicated hallucination evaluation provides a quantitative improvement direction for enhancing LLM reliability.

Limitations of LexBench: Currently focused on the environmental law domain, and evaluation relies on automated metrics. Future directions:

• Expand to other legal branches;
• Introduce subjective evaluations by legal experts to complement quantitative results.