Sherlock: A Self-Correcting Reasoning Framework for Vision-Language Models

Sherlock is the first framework to enable intrinsic self-correcting capabilities in vision-language models (VLMs). Its paper has been accepted by NeurIPS 2025 and open-sourced. The framework achieves significant improvements on multiple benchmarks with only 20K samples. Author: DripNowhy. Project repository link: https://github.com/DripNowhy/Sherlock. Paper link: http://arxiv.org/abs/2505.22651. Released on June 4, 2026.

Vision-language models have made rapid progress in tasks like image understanding and visual question answering, but they face significant bottlenecks in complex reasoning: existing models trained via SFT or RL struggle to perform step-by-step or holistic self-correction. The root cause lies in the training paradigm focusing on single-inference accuracy while ignoring the model's self-verification and correction capabilities. How to enable VLMs to have human-like "reflection-correction" abilities is a key challenge.

The core innovations of the Sherlock framework include: 1. Intrinsic self-correction mechanism without external prompts/multiple sampling; 2. Data efficiency with only 20K samples; 3. Three-stage training process (supervised fine-tuning → offline self-improvement → online self-improvement); 4. Cross-benchmark generalization ability. Built on Llama3.2-Vision-11B-Instruct:

• Stage 1: Supervised fine-tuning with 20K LLaVA-CoT samples to master basic reasoning abilities;
• Stage 2: Offline training with self-generated data to learn error recovery strategies;
• Stage 3: Online construction of 5K preference data using questions and images to dynamically adjust reasoning strategies.

Using the VLMEvalKit evaluation framework, covering tasks like mathematical reasoning, scientific question answering, and visual common sense reasoning. Key findings: Existing VLMs generally lack self-correction capabilities; Sherlock breaks through this bottleneck via its innovative framework and achieves significant improvements. Model weights have been released: https://huggingface.co/collections/Tuwhy/sherlock-6835f46e450a48f228f7e80d.

Implemented with modifications based on LLaMA-Factory and VLMEvalKit:

• Base model: Download Llama3.2-Vision-11B-Instruct (https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct);
• Training data: LLaVA-CoT dataset (https://huggingface.co/datasets/Xkev/LLaVA-CoT-100k). Use 20K samples for SFT/offline stages, and construct 5K preference data using questions + images for the online stage;
• Inference example: Load Sherlock Iter2 weights (Tuwhy/Sherlock-Iter2) and implement inference via the transformers library;
• For training and evaluation guidelines, refer to train/README.md and inference/README.md in the project repository.

The Sherlock paper was accepted by NeurIPS 2025, marking the important academic value of self-correcting reasoning in the VLMs field. The project is fully open-sourced, including training/evaluation code, data construction processes, model weights, and documentation, providing support for community research. Inspiring directions: self-correction mechanism design, data efficiency optimization, and vision-language fusion reasoning.

As the first intrinsic self-correcting VLMs framework, Sherlock achieves performance improvements with a small number of samples through multi-stage training, proving the key role of self-correction in reasoning abilities. Open-source resources will promote the development of VLMs towards more intelligent and reliable reasoning systems, providing a new paradigm for research in related fields.