Fine-tuning Phi-4 for Legal Domain: Specialized Reasoning Practice on the SCOTUS Dataset

This project focuses on the specialized fine-tuning practice of the Phi-4 model in the legal domain. Through training on the SCOTUS 2024 dataset using the LoRA and Unsloth optimization frameworks, it achieves a significant improvement in judicial analysis capabilities (42% increase in F1 score) and provides a complete path for deployment in production environments. This thread will introduce the project background, technical selection, dataset processing, fine-tuning workflow, performance results, deployment solutions, and future outlook in detail across different floors.

The legal industry needs to process massive amounts of professional text, but general-purpose large models perform poorly in legal terminology and case reasoning. Microsoft's Phi-4 model, with its 14 billion parameter scale, efficient reasoning capabilities, 16K long context support, and MIT license-friendly features, has become an ideal base for specialization in the legal domain. This project aims to fine-tune it into a legal expert model and verify its effectiveness on the SCOTUS case dataset.

Technical Selection: Choose LoRA for parameter-efficient fine-tuning (only train <1% of parameters to avoid catastrophic forgetting), combined with the Unsloth optimization framework (2-5x training speedup, 80% memory savings).

SCOTUS Dataset: Contains factual statements, legal issues, court opinions, judgment results, and citation networks of U.S. Supreme Court cases; preprocessing includes structured extraction (separating judge opinions, annotating citations), semantic enhancement (adding concept annotations), and quality control (manual verification).

Training Configuration: Use LoRA rank 64, alpha 128; target modules cover q/k/v/o/gate/up/down proj; training parameters include batch size 2, gradient accumulation 4, 3 epochs, learning rate 2e-4, etc.

Instruction Format: Convert legal tasks into instruction-following format (instruction+input+output) to train the model on structured legal analysis logic.

Multi-stage Training: 1. Legal language adaptation (pre-training on large-scale legal corpora); 2. Task-specific fine-tuning (supervised training on SCOTUS); 3. Preference alignment (DPO optimization for output quality).

Evaluation Metrics: Judgment prediction accuracy, F1 score, legal reasoning quality (precedent citation accuracy, argument logic, etc.).

Key Results: After fine-tuning, the Phi-4-Legal model's F1 score increased from 0.48 to 0.68 (+42%), judgment accuracy from 62% to 78% (+16%), precedent citation accuracy from 45% to 71% (+58%), and legal terminology correctness from 68% to 89% (+31%).

Qualitative Analysis: Improved reasoning depth, more accurate precedent citations, and learned to express legal uncertainty.

Deployment: 1. Ollama integration (Modelfile defines system prompts, one-click startup); 2. GGUF quantization (multi-level versions for different hardware); 3. FastAPI encapsulation of OpenAI-compatible API.

Applicable Scenarios: Legal research assistance, initial contract review screening, education and training.

Limitations: Cannot replace professional lawyers (possible hallucinations), data bias (U.S. law-focused), need to label AI-generated content and include disclaimers.

Insights: Domain specialization is more important than scaling; open-source toolchains (Unsloth, Hugging Face, etc.) lower training thresholds; responsible AI development is needed (boundary statements, hallucination detection).

Future: Expand multi-jurisdiction data, real-time knowledge updates (RAG integration), multi-modal support (contract layout analysis, court hearing audio processing, etc.).