Writing Finetuner: Fine-tuning Mistral 7B on Personal Writing Corpus to Verify if the Model Truly Learns Your Way of Thinking

Writing Finetuner is an open-source project for fine-tuning Mistral7B on personal writing corpora. Its core goal is to address the problem that traditional fine-tuning only focuses on surface metrics like loss and perplexity. It uses innovative reasoning probes to evaluate whether the model truly learns the user's way of thinking, rather than just imitating surface wording habits. The project is maintained by satyam671 and was released on GitHub on June 2, 2026.

The project originated from the author's six-month personal experiment: although the fine-tuned model could imitate writing styles, its reasoning process in new scenarios was significantly different from real thinking. This reveals that standard metrics (such as perplexity and ROUGE-L) can only measure surface text similarity and cannot capture deep reasoning patterns. The project aims to provide a more comprehensive evaluation method to determine whether the model truly learns the user's thinking.

The project provides an end-to-end process with five stages: 1. Corpus preparation (supports Medium HTML, txt, md; automatic cleaning and segmentation); 2. LoRA fine-tuning (parameter-efficient, keeping base model weights unchanged); 3. Standard evaluation (calculating perplexity and ROUGE-L); 4. Reasoning probe (core innovation, evaluating reasoning ability in new scenarios); 5. Deep scoring (structural similarity, reasoning depth, viewpoint consistency). Hardware requirements: LoRA (bf16) requires 24GB VRAM (e.g., RTX3090), fp16 requires 20GB, and with gradient checkpointing enabled, about 16GB; three rounds of training on a 400,000-word corpus take 18-20 hours.

The reasoning probe is a feature of the project, with the design philosophy that 'true understanding is reflected in reasoning ability in new scenarios'. Process: 1. Present an unseen question to the user and the model; 2. The user writes an answer; 3. The model generates an answer; 4. The user blindly evaluates both (to avoid confirmation bias). Example question: 'Diagnostic process for ML models that perform well in testing but degrade in production'. Scoring dimensions: structural similarity, reasoning depth, viewpoint consistency.

Quick start steps: 1. Clone the repository and install dependencies; 2. Place the corpus in data/raw and run the cleaning and chunking script; 3. Configure train_config.yaml; 4. Train; 5. Run the reasoning probe. Typical training results: After three rounds of training, train_loss decreased from 2.143 to 1.203, eval_loss from 2.198 to 1.289, perplexity from 42.3 to 12.1, and ROUGE-L from 0.38 to 0.61. However, the reasoning probe may reveal that the model does not truly understand the way of thinking.

Project insights: Fine-tuning evaluation cannot rely solely on automated metrics; true personalization requires the model to understand the user's thinking rather than copying surface patterns. Summary: Writing Finetuner provides complete tools and methodology, which is of great reference value for users fine-tuning large models on personal data. It is an open-source project that combines technical implementation with profound insights.