AlbabishMeshProject: A Multimodal AI-Powered 3D Medical Mesh Morphological Analysis Tool

This article introduces AlbabishMeshProject—a web-based 3D medical mesh analysis tool that combines classical computer vision with multimodal AI models (such as MedGemma and LLaVA-Med) to provide intelligent support for morphological review of medical image segmentation results. The project aims to help medical researchers and radiologists efficiently review segmentation quality, detect surface abnormalities, and generate structured medical reports.

In the field of medical image processing, 3D reconstruction technology has been widely used in scenarios such as organ modeling and lesion analysis, but the 3D mesh results output by segmentation algorithms are difficult to review quickly. Traditional visualization tools only provide basic rendering and lack the ability for in-depth analysis of morphological features and intelligent interpretation. AlbabishMeshProject emerged as a morphological analysis platform integrating classical CV and multimodal AI to address these pain points.

Frontend Architecture: Based on React framework, supports STL/OBJ formats, real-time rendering and client-side CV analysis (Sobel edge detection, Harris corner recognition, K-means clustering). Backend Architecture: Built with Python Flask, including geometric analysis module (trimesh library for calculating vertex count, surface area, etc.), multi-view rendering engine, and multimodal AI integration (ViT visual embedding, LLaVA-Med medical multimodal understanding, MedGemma for generating structured reports). Morphological Analysis Capabilities: Classical CV feature extraction (edges/contours, corners, region segmentation), mesh geometric property calculation (topological integrity, volume-to-surface area ratio), AI-driven medical semantic interpretation (MedGemma structured reports, LLaVA-Med professional interpretation).

1. Segmentation Quality Review: Automatically check topological integrity and surface smoothness, and mark problematic results; 2. Morphological Abnormality Detection: Detect quantitative indicators such as surface irregularities and branching patterns of tumors/vascular malformations; 3. Anatomical Rationality Verification: Evaluate whether the reconstructed structure conforms to anatomical common sense; 4. Teaching and Research Assistance: Structured reports as teaching materials, and quantitative indicators for disease-morphology correlation research.

1. Modular Design: Separation of front-end and back-end, independent development and testing of each module (e.g., vit_infer.py, llava_med_infer.py in the backend); 2. Local AI Deployment: Deploy MedGemma via Ollama to ensure data privacy, control costs, and support offline use; 3. Progressive Analysis: Front-end lightweight CV for instant feedback → back-end in-depth calculation → AI semantic interpretation, balancing fluency and accuracy.

Current Limitations: Not a diagnostic tool; requires professional review of reports; only supports STL/OBJ formats, pending expansion to native formats like DICOM. Future Directions: Integrate direct analysis of CT/MRI voxel data; support 4D medical image sequence tracking; establish normal anatomical statistical models for abnormality early warning; develop a multi-user collaboration platform.

AlbabishMeshProject is an important attempt to intelligentize and multimodalize medical image analysis tools. It organically combines classical CV, deep learning, and medical knowledge to provide a complete solution for 3D medical mesh morphological review. For developers and researchers in medical image segmentation, computer-aided diagnosis, and anatomical research, it is an open-source project worth paying attention to.