FakaoEval：基于SOLO分类法的大语言模型法律推理能力评估框架

This post introduces the FakaoEval project, which provides evaluation data and code based on the SOLO (Structure of Observed Learning Outcomes) taxonomy to systematically assess large language models' (LLMs) legal reasoning capabilities. Its core goal is to offer a scientific methodology for benchmarking legal AI, moving beyond simple right/wrong judgments to focus on the quality of reasoning.

Legal AI applications are growing (contract review, case retrieval, etc.), but evaluating LLMs' legal reasoning is tricky. Legal reasoning requires understanding complex concepts, applying rules, logical deduction, and compliant conclusions—traditional benchmarks often fail to capture these deep abilities, leading to gaps between evaluation results and real-world performance.

FakaoEval innovates by using SOLO taxonomy (5 levels: prestructural, unistructural, multistructural, relational, extended abstract). The framework includes:

• Dataset: Legal reasoning questions with SOLO level annotations.
• Code: Automated SOLO level判定 logic.
• Multi-dim analysis: Focuses on reasoning structure (e.g., multiple legal elements, their relationships, abstraction).

Key technical components:

• Prompt engineering: Structured prompts to elicit reasoning processes (not just conclusions).
• Output parsing: Analyze text structure to identify arguments, evidence, and relationships.
• Level classifier: Rule-based or ML models to determine SOLO levels.
• Comparison: Parallel evaluation of multiple models and detailed reports.

FakaoEval contributes:

1. Shift from result to process: Provides fine-grained feedback for model improvement.
2. Cross-domain application: Applies educational assessment theory to AI evaluation.
3. Diagnostic tool: Identifies model weaknesses in legal reasoning.
4. Benchmark reference: Offers an extensible framework for legal AI benchmarks.

Use cases include:

• Model selection: Evaluate commercial/open-source models for specific tasks.
• Capability tracking: Measure improvements across model iterations.
• Error analysis: Locate weak points via SOLO level of failed cases.
• Training guidance: Design targeted training data/strategies based on results.

Current limitations and future work:

• Subjectivity: SOLO level判定 needs more standardization and human validation.
• Language: Focus on Chinese legal text; cross-language applicability needs testing.
• Dynamic updates: Dataset needs regular updates to reflect evolving legal practices.
• Indicator correlation: Study links between SOLO levels and traditional metrics (accuracy, F1).

FakaoEval innovatively applies SOLO taxonomy to LLM legal reasoning evaluation, focusing on reasoning quality rather than just correctness. It provides a new methodological perspective for legal AI assessment and improvement, promoting the scientific development of legal artificial intelligence.