WamGLM: An Intelligent Wafer Defect Analysis System Based on Multimodal Large Language Models

WamGLM is an open-source multimodal large language model designed for semiconductor wafer defect detection scenarios. It enables multi-turn conversational deep queries through prototype-supervised contrastive learning, combining computer vision and natural language processing technologies to solve the problems of low efficiency and difficult knowledge inheritance in traditional defect analysis, thus providing an intelligent solution for quality control in chip manufacturing.

In modern semiconductor manufacturing, wafer defect detection is a core yield-related process. However, shrinking process nodes lead to complex defect types, and traditional experience-dependent methods are limited in efficiency when dealing with massive data, making it difficult to systematically inherit knowledge. Wafer maps contain rich spatial information, but associating them with historical data and process parameters for analysis is challenging. The rise of multimodal large language models has brought new solutions to this field.

WamGLM adopts prototype-supervised contrastive learning to construct prototype representations of defect categories, narrowing the distance between samples of the same category and increasing the distance between different categories to enhance generalization ability. Its multimodal fusion architecture extracts features from wafer maps via a visual encoder, understands text queries via a language encoder, deeply fuses them through an attention mechanism, and achieves multi-turn conversational interaction with context management.

WamGLM can be applied in process development (quickly analyzing test wafer defects and providing consistent evaluation standards), production monitoring (integrating with MES for real-time analysis, anomaly warning, and diagnosis), and knowledge management (converting conversation records into a knowledge base to accelerate new employee training). It reduces subjective bias and rework costs.

Currently, WamGLM has limitations such as insufficient scale and diversity of training data, limited coverage of specific defects, and being affected by map resolution and annotation quality. In the future, it will integrate multi-source sensor data, develop an efficient inference engine, and establish a defect knowledge graph to meet the needs of real-time performance and complex reasoning.

WamGLM combines multimodal learning theory with semiconductor process requirements, serving as a typical case of AI empowering quality control in traditional industries. It provides an open-source project worth researching and referencing for the fields of intelligent manufacturing and industrial AI.