Innovative Application of Multimodal Vision-Language Models in Building Entrance Detection

This article introduces a multimodal building entrance detection system that integrates aerial imagery, street view images, GPS trajectories, and geospatial data. By fine-tuning vision-language models using LoRA and DoRA technologies, it achieves accurate spatial reasoning and positioning, addressing the problem of limited detection accuracy in traditional single-data-source methods. It has practical value in scenarios such as intelligent navigation and emergency rescue.

Accurately locating building entrances is extremely challenging in scenarios like urban navigation and emergency rescue. Traditional methods rely on single data sources (e.g., satellite images or street views), which are susceptible to factors such as occlusion and lighting, leading to limited accuracy. With the development of multimodal learning, integrating multiple data sources has become an approach to improve detection performance, and this project builds a comprehensive multimodal entrance detection system.

The core innovation lies in the integration of four types of data: aerial imagery (overhead layout), street view images (ground details), GPS trajectories (human movement patterns), and geospatial data (building outlines/road networks). Vision-language models (VLMs) are used as the foundational architecture, and their cross-modal understanding capabilities are suitable for spatial reasoning tasks.

Two fine-tuning technologies, LoRA and DoRA, are used: LoRA fine-tunes attention layers by injecting low-rank matrices, achieving full-parameter fine-tuning results with a small number of trained parameters and reducing resource requirements; DoRA is an improved version of LoRA that decomposes weights into magnitude and direction components for separate fine-tuning, enhancing performance while maintaining parameter efficiency.

The project adopts a modular design. The src directory contains components such as baseline models (e.g., random forests), ViT+LoRA implementations, and data loading modules; it provides EDA Notebooks, training/evaluation scripts; uses Miniconda for environment management, and configures pre-commit hooks for automated format checking, reflecting good engineering practices.

The system has wide applications in multiple fields: intelligent navigation improves the last-mile experience; emergency rescue facilitates rapid deployment; logistics delivery optimizes routes; it can also provide entrance distribution data for urban planning to assist in infrastructure optimization.

The project demonstrates the potential of multimodal learning in geospatial tasks. Integrating heterogeneous data with parameter-efficient fine-tuning can achieve high-performance models under resource constraints. In the future, with the evolution of VLMs and the reduction of data costs, such methods are expected to promote progress in smart cities, autonomous driving, and other fields.