RefDiff: A Fine-Grained Industrial Anomaly Detection Framework Based on Multimodal Large Language Models

RefDiff is an innovative reference-conditioned difference framework that draws on the LLaVA architecture, applying multimodal large language models to the field of industrial anomaly detection to achieve more precise fine-grained defect recognition. As an open-source project, its core lies in combining multimodal models with difference learning and introducing reference images as conditions to enhance detection accuracy and interpretability.

Anomaly detection in industrial manufacturing is an important topic in computer vision. Traditional methods face challenges such as difficulty handling complex scenarios, insufficient fine-grained defect recognition, and lack of effective reference comparison mechanisms. The development of multimodal large language models (MLLMs) provides a new direction for their migration to the industrial detection field.

RefDiff is an open-source reference-conditioned difference framework inspired by the LLaVA architecture. Its core innovation is combining multimodal large language models with difference learning and introducing reference images as conditions. The design follows a three-stage process of "Reference-Difference-Judgment": receiving the image to be detected and the reference image → extracting feature differences → using large language models to infer and determine defects, making full use of MLLMs' visual understanding and language reasoning capabilities.

Multimodal Feature Extraction

Adopts a collaborative architecture of visual encoder and language model: the visual encoder extracts high-level semantic features of images, while the language model is responsible for reasoning and interpretation, enabling both the identification of abnormal regions and the generation of understandable anomaly descriptions.

Reference Condition Mechanism

As a core innovation, it introduces reference images as additional conditional inputs. By calculating the difference features between the image to be detected and the reference image, it can more accurately locate abnormal regions and distinguish between real defects and normal image changes.

Difference Learning Strategy

Adopts a fine-grained feature comparison strategy, focusing on global differences while capturing local subtle anomaly patterns, which is suitable for detecting industrial defects such as tiny texture changes and local geometric deformations.

Industrial Quality Inspection Scenarios

Applicable to production line quality inspection scenarios such as electronic component detection (identifying soldering defects, scratches, stains) and textile detection (finding weaving defects or uneven dyeing).

Fine-Grained Recognition Capability

Compared with traditional methods, it can accurately locate abnormal regions and generate detailed descriptions (e.g., "There is a 2mm scratch in the upper left corner") instead of only providing an anomaly score.

Enhanced Interpretability

By introducing a language model component, the detection results are interpretable: it not only informs about anomalies but also explains the causes and specific manifestations, helping quality inspectors understand and trust the AI results.

As an open-source project, RefDiff's code is publicly available on GitHub, providing valuable resources for research and applications in the industrial anomaly detection field. Researchers and engineers can conduct secondary development to adapt to specific scenarios, and its LLaVA-style architecture also provides a reference paradigm for other multimodal industrial AI applications.

With the development of multimodal large language models, RefDiff is expected to be applied to more industrial scenarios. Future directions include: supporting more industrial data types such as 3D point clouds and infrared images; achieving real-time detection to meet production line speed requirements; and developing lightweight models to adapt to edge computing scenarios.