WamGLM: A Multimodal Large Language Model for Wafer Defect Detection

WamGLM is a multimodal large language model designed for wafer defect detection. It combines prototype-supervised contrastive learning with a multi-turn dialogue framework to achieve end-to-end recognition of wafer map defects and in-depth information querying, demonstrating professional application potential in the field of semiconductor manufacturing quality control. Its core innovation lies in the deep integration of visual defect recognition and natural language question-answering capabilities, providing methodological references for industrial AI applications.

Semiconductor manufacturing is a precision capital-intensive industry, where wafer quality determines product yield. Although traditional deep learning visual models can classify defects, they struggle to handle dynamic queries (such as defect types, causes, process adjustments, batch correlations, etc.). These issues require models to have image understanding, process knowledge association, and multi-turn reasoning capabilities, which multimodal large language models (MLLMs) have the potential to provide.

Technical Architecture

WamGLM adopts an end-to-end multimodal architecture: the visual encoder extracts features from wafer maps, the cross-modal projection layer aligns with the language space, and the language model backbone generates responses, avoiding the accumulation of pipeline errors.

Prototype-Supervised Contrastive Learning (PSCL)

To address defect category imbalance and intra-class diversity, PSCL learns category prototype vectors and optimizes intra-class compactness and inter-class separability through contrastive loss, enhancing feature discriminability.

WaferMapVMQA Dataset

The first large-scale multi-turn question-answering dataset for wafer defects was constructed. Professional dialogues are generated through LLM interaction and manually reviewed to ensure quality, covering scenarios such as defect classification and cause analysis.

Two-Stage Training

1. Visual fine-tuning: Use PSCL to optimize defect features; 2. Language fine-tuning: Use WaferMapVMQA to train multi-turn dialogue capabilities, with separate training to avoid interference.

Validated on real wafer datasets:

• Defect Recognition: Outperforms existing methods; PSCL enhances the recognition ability of rare defects and variants;
• Information Query: Accurately understands intentions and handles complex traceability queries through multi-turn dialogue;
• Ablation Experiments: Removing PSCL leads to a decrease in accuracy, and skipping visual fine-tuning results in insufficient image understanding, verifying the effectiveness of the strategy.

• Online Quality Monitoring: Integrate with equipment to recognize defects in real time, answer operators' queries in natural language, reducing professional dependency;
• Defect Root Cause Analysis: Trace causes through multi-turn queries, associate process parameters to accelerate problem-solving;
• Knowledge Inheritance: New employees learn defect knowledge through interaction, improving training efficiency;
• Historical Data Mining: Query historical batches in natural language to discover quality trends and optimization opportunities.

• Domain-Specific Models: General models struggle to meet industrial precision requirements; domain models improve performance through targeted data and training;
• Universality of Prototype Learning: PSCL is suitable for visual tasks with intra-class diversity and data imbalance;
• Dialogue Data Construction: Generating domain dialogues via LLM interaction can be extended to other vertical fields, reducing data costs;
• Trinity Integration: Deep integration of visual perception, language interaction, and domain knowledge is key to industrial AI.

Limitations: Only targets wafer maps; generalization to SEM images and others remains to be verified; limited context length; poor performance on rare defects. Future Directions: Expand support for more semiconductor images; introduce retrieval-augmented generation to connect to real-time databases; develop model interpretability; explore lightweight edge deployment.

WamGLM improves visual recognition through PSCL, constructs a multi-turn dialogue dataset to inject domain knowledge, and achieves end-to-end recognition and in-depth querying of wafer defects. It provides a new tool for semiconductor quality control and also offers methodologies for industrial multimodal AI applications: domain-specific models, targeted training strategies, and high-quality datasets are key to transforming general AI into industrial value.