OmniSch: A Specialized Benchmark for Evaluating Large Models' PCB Schematic Understanding Capabilities

In Electronic Design Automation (EDA), converting PCB schematics to netlists requires complex visual understanding, topological analysis, and geometric reasoning, but existing multimodal large models have obvious shortcomings in this task. The OmniSch benchmark includes 1854 real schematics and four evaluation tasks, revealing the significant deficiencies of models in fine-grained visual localization, topological relationship understanding, geometric reasoning, etc., and providing a standardized evaluation platform for this field.

PCB schematics are a key bridge connecting design intent and physical implementation, and EDA software needs to convert them into netlist data. Multimodal large models have made significant progress in general tasks, but their ability to understand professional engineering drawings lacks systematic evaluation. The challenges of PCB schematics include: highly symbolic visual language requiring semantic understanding, core value lying in global reasoning of topological relationships, strong geometric attributes requiring precise spatial information capture, and large scale and complex structure of real schematics requiring handling of long-distance dependencies.

The OmniSch dataset contains 1854 PCB schematics from real engineering projects, covering different complexities. Based on this, four progressive tasks are designed:

1. Visual Localization and Entity Recognition: Annotate 109,900 device instances to evaluate localization accuracy and classification accuracy;
2. Graph Structure Reasoning and Topological Understanding: Convert schematics into netlist graph structures and identify connection relationships;
3. Geometric Reasoning and Spatial Weight Calculation: Extract geometric features of connections (length, direction, etc.);
4. Tool-Enhanced Agent Reasoning: Allow models to call external tools to complete tasks, simulating the working method of human engineers.

Evaluation results of each task:

• Visual Localization and Entity Recognition: Coarse-grained recognition is acceptable, but fine-grained localization is insufficient (inaccurate bounding boxes, confusion of adjacent components);
• Graph Structure Reasoning and Topological Understanding: Local connection recognition is acceptable, but global connectivity reasoning is poor (errors in cross-page network label tracking);
• Geometric Reasoning and Spatial Weight Calculation: The weakest performance, with frequent errors in simple distance calculation or direction judgment;
• Tool-Enhanced Agent Reasoning: Even with tool assistance, there are still inefficient visual explorations (blind global search, resource waste).

The OmniSch evaluation reveals significant gaps in existing multimodal large models' understanding of professional engineering drawings:

1. Unreliable fine-grained localization: Inaccurate bounding boxes for small components and improper handling of overlapping areas;
2. Fragile layout-to-graph conversion: Errors are prone in converting complex intersections, bus structures, and cross-page connections;
3. Inconsistent global connectivity reasoning: Locally correct but globally contradictory;
4. Inefficient visual exploration: Lack of effective strategies during active search, leading to missing key information.

Insights from OmniSch for multimodal model development:

1. Build domain-specific training and evaluation data (general datasets cannot cover the particularities of engineering drawings);
2. Strengthen geometric reasoning capabilities (introduce explicit geometric reasoning modules);
3. Deeply explore tool usage and agent reasoning (simulate the tool-assisted methods of human engineers);
4. Innovate evaluation metrics (develop metrics more aligned with engineering application needs).

Industrial value of automated PCB schematic understanding:

• Lower EDA entry barriers: Replace expensive specialized software to achieve low-cost and easy-to-use schematic-to-netlist solutions;
• Expand application scenarios: Design review, document generation, knowledge management (e.g., automatic error checking, design document generation, extraction of reusable modules);
• Promote technological progress: The standardized evaluation platform helps researchers objectively compare the effects of methods and accelerate technological iteration.