Hands-On Fine-Tuning of Mistral Large Model: Complete Workflow and Effect Evaluation Based on LlamaIndex

This article introduces an open-source project developed by the botextractai team, showing how to perform domain-specific fine-tuning on open-mistral-7b using LlamaIndex and MistralAI API (with IPCC climate reports as data), and quantifies the performance improvement before and after fine-tuning via the Ragas evaluation framework. The project provides an end-to-end complete workflow from data processing to effect evaluation, which is of great reference value for learning LLM fine-tuning techniques.

The project aims to provide a reproducible Mistral model fine-tuning workflow. It selects open-mistral-7b as the base model and uses Chapter 3 of the IPCC Sixth Assessment Report (WGII) as domain data, with the goal of improving the model's performance in climate science question-answering tasks. Its unique feature lies in end-to-end completeness—each link (document processing, data generation, fine-tuning, evaluation) has clear code implementation and result records.

The project integrates mainstream tools to form a fine-tuning pipeline:

• MistralAI API: Provides base models and fine-tuning APIs, simplifying interactions via MistralAIFinetuneEngine;
• LlamaIndex: Responsible for PDF loading, chunking, and index construction, enabling conversion of documents to training data;
• Weights & Biases (W&B): Monitors the training process and records experimental metrics;
• Ragas: Evaluates RAG systems and provides answer relevance and faithfulness metrics;
• OpenAI API: Generates synthetic question-answer pairs and calculates evaluation metrics.

The data source is Chapter 3 of the IPCC Sixth Assessment Report (WGII). Data generation is divided into two phases:

1. Use mistral-small-latest to generate 40 training questions and 40 evaluation questions from different chapters of the document (to avoid data leakage);
2. Use mistral-large-latest to generate high-quality synthetic question-answer pairs, outputting as training.jsonl. The strategy of 'strong model generation, weak model learning' is adopted to quickly build a domain training set.

Fine-tuning is executed via LlamaIndex's MistralAIFinetuneEngine. Only the training data and base model name need to be provided, and it automatically handles uploads, training, and progress monitoring. During training, W&B records real-time metrics such as loss curves and learning rates for easy debugging. After fine-tuning is completed, a model ID (in the format ft:open-mistral-7b:...) is returned for subsequent calls.

Evaluate the performance before and after fine-tuning using Ragas:

• Answer Relevance: Measures the relevance of the answer to the question, higher is better;
• Faithfulness: Measures the factual consistency between the answer and the context, higher is better. Results: Before fine-tuning (open-mistral-7b): relevance 0.825, faithfulness 0.930; After fine-tuning: relevance 0.844 (+2.4%), faithfulness 0.964 (+3.6%). Although the improvement magnitude is not large, the improvement on a high baseline has practical value and provides data support for fine-tuning decisions.

Applicable Scenarios:

• AI developers learning the complete LLM fine-tuning workflow;
• Application developers needing to improve model performance in specific domains (law, medical, climate);
• Technical teams understanding RAG evaluation;
• MistralAI ecosystem researchers. Learning Value: The end-to-end workflow demonstration emphasizes scientific evaluation (rather than just running through code), which is more critical for actual production.

Explorable expansion directions:

• Try different base models (larger Mistral versions or other open-source models);
• Use more evaluation metrics (BLEU, ROUGE, BERTScore, etc.);
• Compare few-shot and full fine-tuning;
• Integrate more data sources to build large-scale training sets;
• Add manual evaluation to complement automatic metrics.