Evaluation of Large Language Models on Vietnamese Legal Texts: From Benchmark Testing to Reasoning Ability Analysis

This article conducts a comprehensive evaluation of four large language models—GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1—on the task of simplifying Vietnamese legal texts. Using a dual evaluation framework (quantitative performance benchmarking + qualitative error analysis), it reveals the trade-off between accuracy, readability, and consistency among the models, identifies the core challenge of current LLMs as insufficient legal reasoning ability, and proposes methodological contributions and practical implications.

The complexity of legal texts hinders public access to judicial justice. Vietnamese legal texts are known for their technical language, complex structure, and dense terminology. LLMs bring hope for simplification, but traditional metrics (BLEU/ROUGE) fail to capture key dimensions of legal applications (accuracy, readability, consistency) and make it difficult to explain error causes.

The dual evaluation framework includes:

1. Three-dimensional performance benchmarking: Evaluates accuracy (semantic fidelity), readability (Vietnamese-specific metrics + reader tests), and consistency (terminology stability), involving 4 advanced LLMs;
2. Large-scale error analysis: Based on a dataset of 60 Vietnamese legal provisions, uses an expert-validated classification system (misinterpretation, incorrect examples, etc.) to analyze error types.

1. Performance trade-offs: Grok-1 excels in readability/consistency but has low accuracy; Claude 3 Opus has high accuracy but hides reasoning errors; GPT-4o/Gemini 1.5 Pro are balanced but have no outstanding advantages;
2. Reasoning challenges: The core issue is controlled and accurate legal reasoning (complex logic, lack of domain knowledge, failure to capture subtle semantic differences);
3. Error distribution: Misinterpretation errors account for the highest proportion, followed by incorrect example errors.

1. Vietnamese legal benchmark dataset: 60 multi-domain provisions, including original texts, expert-simplified versions, and annotations;
2. Expert-validated error classification: A structured framework for automated detection and manual review;
3. General framework: Can be applied to text simplification evaluation in other languages/professional fields.

Development implications: Beware of the trap of surface fluency, prioritize error analysis over overall metrics, and adopt human-machine collaboration models; Technical directions: Domain-adaptive training (continued pre-training/RAG), reasoning enhancement (chain-of-thought/multi-round verification), legal-specialized RLHF; Expansion: The framework can be applied to other legal systems (civil/common law).

The study goes beyond surface performance to deeply understand the limitations of LLM legal reasoning. Current LLMs have systemic deficiencies in core reasoning abilities; future breakthroughs need to focus on understanding the essence of legal reasoning and targeted technical design. Developers should attach importance to error cause analysis and build reliable legal AI systems.