COLLT: A Clarification-Oriented Tool Learning Framework for Legal Large Language Models

COLLT (Clarification-Oriented Legal Language with Tool augmentation) is a clarification-oriented tool learning framework specifically designed for the legal domain. Its core goal is to address the common information gap in user legal consultations and improve the response quality of large models in complex legal scenarios. The framework features an intelligent clarification mechanism and six professional legal tools. It simulates real lawyers' workflow through a dual-track decision-making mechanism, uses budget control to prevent tool abuse, supports low-resource training and multi-model adaptation, and has verified its effectiveness through rigorous evaluations and open-sourced related resources.

On Chinese online legal service platforms, user consultations often suffer from severe information insufficiency (e.g., only asking "Can I get a divorce?" without specifying key information like marriage duration or property status). When faced with ambiguous queries, traditional large models either give vague answers or make inferences based on assumptions, leading to untargeted or even misleading suggestions. COLLT is proposed to address this pain point, empowering models with the ability to intelligently judge when to clarify and call professional tools to obtain authoritative evidence.

The core innovation of COLLT is its action tagging system: indicates the need to clarify information, and indicates that it can directly enter the tool retrieval and response phase. The framework has six built-in professional tools trained on Lawformer: T_LAS (Legal Article Search), T_LCP (Crime Prediction), T_SCR (Similar Case Retrieval), T_LER (Element Recognition), T_LED (Event Detection), and T_NET (Web Search). Additionally, Proposition 1 restricts each dialogue round to a maximum of 2 tool calls to avoid excessive retrieval and achieve precise tool combinations (e.g., T_LER and T_LAS are prioritized for divorce property disputes).

COLLT uses 4-bit QLoRA technology to successfully fine-tune five mainstream Chinese large models such as ChatGLM3-6B and LLaMA-3-8B on a single NVIDIA RTX 4090 (24GB VRAM). The low-resource solution is easy for small and medium-sized teams to reproduce. The training data comes from 11,533 real consultation seeds from DISC-Law-SFT. After two-stage annotation by DeepSeek (judging clarification needs and tool calls), the COLLT-SFT dataset (in OpenAI message format) containing 11,528 multi-turn dialogues was generated.

COLLT's evaluation covers nine regular LawBench tasks and the specially constructed AmbigLegalQA evaluation set (5,181 samples covering 0-4 rounds of clarification). Evaluation metrics include Trigger-F1 (accuracy of clarification judgment), Clarification Coverage (whether clarification questions cover key gaps), and Multi-turn ROUGE-L (response matching degree). Ablation experiments show that the clarification mechanism and tool system can only exert maximum effectiveness when working in synergy.

The COLLT project open-sources training code, evaluation scripts, dataset construction processes, and 11,528 training samples (under CC BY-NC 4.0 license), filling the gap of scarce high-quality Chinese legal dialogue datasets. Its explicit protocol tagging system (e.g., , ) provides a reference paradigm for the tool learning field, facilitating debugging, interpretability analysis, and subsequent research interventions.

Insights from COLLT for legal AI developers: 1. Domain-specific tool design is crucial—general tools cannot meet the precision requirements of law; 2. The clarification mechanism improves user experience and trust; 3. The budget control strategy can be applied to other tool learning scenarios. COLLT's open-source resources lay a solid foundation for the further development of legal AI.