CAVG: A New Scheme for Autonomous Driving Visual Grounding Integrating GPT-4 and Cross-Modal Attention Mechanism

This article introduces the CAVG (Context-Aware Visual Grounding) model, which integrates the GPT-4 large language model and a five-encoder architecture to achieve high-precision multimodal visual grounding in autonomous driving scenarios, achieving SOTA performance on the Talk2Car dataset. Its core innovation lies in combining GPT-4's semantic understanding capability with cross-modal attention mechanism to solve the key problem of mapping natural language instructions to target objects in visual scenes.

One of the core goals of autonomous driving is natural and efficient human-vehicle interaction. The visual grounding task requires mapping natural language instructions to specific targets in visual scenes. This task faces multiple challenges: natural language contains rich context and emotions, and simple keyword matching is difficult to capture deep intentions; real traffic scenes have bad weather, occlusion, lighting changes, and multi-target interference; the system has extremely high requirements for real-time performance and accuracy, and misjudgments can easily lead to safety hazards.

The CAVG model adopts a five-encoder architecture: the text encoder converts instructions into vectors; the emotion encoder captures the emotional color of instructions (such as urgency); the visual encoder processes images to generate Region of Interest (RoI) representations; the context encoder injects scene context into RoIs; the cross-modal encoder fuses text, emotion, and visual information through multi-head attention; the multimodal decoder uses a Region-Specific Dynamic layer to calculate matching scores and select the optimal region.

The innovations of CAVG include: 1. Hybrid strategy context analysis, where text and visual information interact deeply instead of simple late fusion; 2. Integration of GPT-4 to achieve emotional understanding, capturing subtle emotions in instructions to adapt to different response needs; 3. Strong robustness and generalization ability, stable performance in bad weather, complex instructions, and crowded scenes, and good generalization even with limited training data.

In the evaluation on the Talk2Car benchmark dataset, CAVG achieved an average precision (AP50) of 74.55% at IoU=0.5, surpassing all SOTA methods. The previous best method FA was 73.51%, and the early baseline method STACK-NMN was only 33.71%. This proves the effectiveness of its architectural design and marks the transition of visual grounding technology from simple multimodal fusion to deep semantic understanding.

The practical value of CAVG includes: 1. Improving passenger experience, enabling vehicles to respond to human instructions more naturally, and facilitating human-machine collaboration in shared mobility; 2. Providing a hybrid architecture paradigm of "large model + professional modules" to serve as a reference for multimodal AI applications; 3. Lowering the development threshold, achieving high performance even with limited training data, which is beneficial for R&D by teams with limited resources.

CAVG represents an important progress in autonomous driving visual grounding technology, integrating GPT-4 and cross-modal attention mechanism to achieve deep understanding and precise positioning. Looking forward, the continuous development of large language models and multimodal technologies will promote more intelligent and natural human-machine interaction systems, and CAVG's paradigm of "deep semantic understanding + precise visual grounding" may become a standard configuration for the next generation of autonomous driving.