Practical Guide to Fine-Tuning Mistral Large Model: A Complete Workflow with LlamaIndex and Weights & Biases

This article introduces a practical project for fine-tuning Mistral's open-source model based on LlamaIndex and Weights & Biases (W&B), covering the complete workflow of data preparation, baseline evaluation, synthetic data generation, training monitoring, and performance comparison, to help general-purpose models adapt to domain-specific tasks.

The project targets Mistral AI's open-mistral-7b model for fine-tuning. Core technical components include: the base model open-mistral-7b, LlamaIndex's MistralAIFinetuneEngine wrapper, W&B experiment tracking, Ragas evaluation library, and mistral-small/large for synthetic data generation. The advantages of this combination are LlamaIndex simplifying code, W&B enabling real-time monitoring, and Ragas providing professional RAG evaluation capabilities.

Using Chapter 3 of the IPCC Sixth Assessment Report PDF as the knowledge source: training data consists of 40 Q&A pairs generated by mistral-small-latest, and evaluation data consists of 40 Q&A pairs from different chapters generated by mistral-large-latest. Both are saved in JSONL format to avoid data leakage and ensure generalization ability.

Before fine-tuning, a baseline evaluation is performed on the original model using the Ragas library, focusing on two metrics: Answer Relevance (baseline: 0.8248) which measures the match between the answer and the question, and Faithfulness (baseline: 0.9297) which measures the factual consistency between the answer and the context, providing a reference for fine-tuning results.

Fine-tuning is completed via the MistralAIFinetuneEngine wrapper. W&B records key metrics like loss curves and learning rates in real time, enabling visual monitoring of the training process. After fine-tuning, a model ID is obtained for subsequent evaluation and invocation.

Performance improvements after fine-tuning: Answer Relevance increased from 0.8248 to 0.8443 (+2.36%), Faithfulness increased from 0.9297 to 0.9635 (+3.64%). The improvement in Faithfulness is more significant, and this was achieved with only 40 synthetic samples. Increasing data volume or iterative optimization may yield more obvious results.

API keys are recommended to be stored in .env files or environment variables; the llama-index-finetuning package of LlamaIndex requires manual modification of utils.py to resolve dependency issues; timely cleanup of unused fine-tuned models to control costs.

This project demonstrates a lightweight and complete LLM fine-tuning workflow, which is valuable for teams looking to quickly validate fine-tuning effects. Future directions include exploring larger data scales, trying different base models, manual review of synthetic data, and Parameter-Efficient Fine-Tuning (PEFT) to reduce costs.