Empirical Study on Legal Clause Classification Using Instruction-Tuned LLMs

The open-source project by the master's team from the University of Liverpool explores how to use instruction-tuned large language models such as Phi-3-mini to implement binary classification tasks for legal clauses through prompt engineering, and evaluates their application potential in the field of legal NLP. The project focuses on the effectiveness of lightweight models in resource-constrained environments, providing a reproducible benchmark for legal AI applications.

Legal document processing is a highly specialized and time-consuming task; manual identification of the nature of key clauses (binary classification) is costly and inefficient. General-purpose LLMs face challenges in legal applications: the uniqueness of legal language, the professionalism of terminology, and strict requirements for accuracy. The master's team from the University of Liverpool conducted an empirical study to address this need.

Research Objectives: Evaluate the performance of instruction-tuned LLMs in binary classification tasks for legal clauses, and explore the feasibility of lightweight models. Model Selection: Phi-3-mini-4k-instruct (3B parameters). Its advantages include high parameter efficiency, instruction-tuned version suitable for prompt tasks, sufficient 4K context, and open-source accessibility. Tech Stack: HuggingFace Transformers framework, float16 half-precision inference, greedy decoding strategy (output length limited to 5 tokens). Prompt Engineering: Structured templates to set the role of legal experts, constrain output to only Yes/No, and use concise input-output structure. Data Processing: TSV format (text column stores clauses, answer/label column stores labels), labels uniformly mapped to binary values (Yes=1/No=0).

Evaluation Metrics: Accuracy (overall performance) and F1 score (harmonizes precision and recall, suitable for class imbalance scenarios). Project Structure: Clear directory structure (data/, scripts/, experiments/, results/) for easy reproducibility. Code Management: Git workflow (main branch for stability, feature/* branches for independent tasks, PR merging to ensure quality).

Application Value: Provides a lightweight and reproducible benchmark for legal text classification, demonstrating the feasibility of small-parameter models, the importance of prompt engineering, and a path for lightweight deployment. Limitations: Only supports binary classification; 3B models may struggle with complex legal logic; lacks domain adaptation; limited evaluation dimensions (does not cover interpretability/fairness).

Future Directions: Expand to multi-label classification; test larger models; integrate RAG to enhance factual accuracy; domain-specific fine-tuning for legal fields; research on interpretability. Conclusion: The project fully demonstrates the application process of LLMs in professional fields, provides a reference for legal NLP beginners, and its open-source release promotes community iteration and the democratization of legal AI.