SemCath: Real-Time 3D Reconstruction Technology for Interventional Surgery Driven by Multimodal Large Language Models

SemCath achieves real-time reconstruction from 2D fluoroscopy images to 3D anatomical structures by combining medical reasoning and neural rendering, providing intelligent navigation support for cardiovascular interventional surgery. Its core innovation lies in transforming the geometric inverse problem of traditional 3D reconstruction into a medical reasoning problem, and introducing multimodal large language models to enhance the ability to understand the anatomical implications behind the images.

In cardiovascular interventional surgery, traditional 2D fluoroscopy images only provide planar projection information, making it difficult to obtain accurate 3D vascular structures in real time, which is a core challenge in the field of medical imaging. SemCath proposes a new idea: redefining 3D reconstruction as a medical reasoning problem rather than a traditional geometric inverse problem. With the semantic understanding ability of multimodal large language models, AI can 'read' the anatomical meaning of images like an experienced doctor.

The SemCath system consists of three modules forming a complete pipeline:

1. Medical Scene Understanding Module (MSU)：Uses a medically adapted multimodal large language model to extract high-level semantic information from 2D fluoroscopy sequences (e.g., vascular morphology, lesion characteristics), and integrates medical knowledge graphs to associate image features with anatomical knowledge;
2. Semantic-to-Geometric Translation Engine：Maps clinical concepts (e.g., "moderate stenosis in the middle segment of the left anterior descending artery") to parameterized 3D geometric primitives, ensuring anatomical rationality under medical constraints such as Murray's law and vascular topology rules;
3. Adaptive Neural Rendering System：Models data noise and model confidence through variational inference, generating high-quality 3D anatomical models weighted by confidence.

SemCath was compared with 9 baseline methods on the SOFA simulation platform dataset, and statistical significance was verified through 5-fold cross-validation and paired Wilcoxon signed-rank test (Bonferroni correction, p<0.01):

• Pathological recognition rate increased by 27% (0.623→0.791), which is of significant value for surgical navigation;
• Anatomical consistency +9.2%, centerline deviation -14.9%, volume overlap rate +4.9%;
• Inference time is 278.3ms, meeting clinical real-time requirements.

Training and evaluation are based on a simulation platform built with the SOFA framework, which has the following features:

• Patient-specific vascular geometry (from clinical CT angiography);
• Physiologically calibrated biomechanics (Young's modulus 1.0-8.0MPa, cardiac displacement 3-8mm);
• Diverse pathological manifestations (stenosis 30-90%, calcification density 800-2000HU);
• Realistic imaging chain simulation (polychromatic X-ray spectrum, scattering modeling, dynamic distribution of contrast agent), ensuring the transferability of the model to clinical scenarios.

Core advantages of SemCath over traditional methods:

• Semantically guided reconstruction is more in line with anatomical common sense, reducing unreasonable artifacts;
• Output confidence information helps doctors evaluate the reliability of results;
• Natural language interface lays the foundation for future interactive surgical navigation.

With the continuous progress of multimodal large language models in the medical field, semantic-driven methods like SemCath are expected to expand to more clinical scenarios, promoting the development of intelligent surgical navigation technology to a higher level.