Visual Evidence Tracing for Multimodal Large Models: Interpretability Challenges in Autonomous Driving Scenarios

The study focuses on the visual evidence tracing problem of multimodal large models in autonomous driving scenarios, proposing a multi-view visual question answering benchmark that requires models to identify the correct camera view supporting the answer. Experiments found that models often give correct answers but based on incorrect visual evidence, exposing the grounding flaws of multimodal models, which has important warning implications for safety-critical applications.

Multimodal Large Language Models (MLLMs) have achieved impressive results in visual reasoning benchmarks, but a core issue is overlooked: does the model really 'look' at the right place when giving a correct answer? In autonomous driving multi-view scenarios, vehicles are equipped with multiple cameras (e.g., six synchronized views in the NuScenes dataset). Models may guess the correct answer based on wrong views (such as reflections/shadows from side-view cameras). While these answers are indistinguishable at the answer level, the safety implications are vastly different.

Benchmark Design

The study constructs a multi-view visual question answering benchmark. Core task: Given six synchronized camera views from NuScenes and a question, the model must simultaneously identify the correct camera view and answer the question. Data construction uses automatic conflict mining + manual verification, containing 122 conflicting question-answer pairs (73 scenarios, covering causal/counterfactual reasoning and other types), ensuring each sample has a clear 'golden view'.

Evaluation Setup

1. View Selection Setup: Evaluate only the ability to select the correct camera view;
2. Oracle QA Setup: Assume the golden view is known, evaluate the QA ability under that view;
3. Joint Prediction Setup: Select the view and answer the question simultaneously (closest to real-world applications).

Answer evaluation: Exact match for structured answers; LLM-based judgment for open-ended answers.

The benchmark explicitly separates visual source identification from answer correctness, exposing grounding failures that cannot be detected by answer-only evaluation: Models may give correct answers in joint prediction, but the selected view has no causal relationship with the answer—meaning the model makes 'informed guesses' rather than true visual reasoning.

The study warns: In safety-critical applications like autonomous driving, we cannot trust decisions just because models perform well on test sets; we must ensure decisions are based on correct visual evidence.

Future research directions:

1. Develop multimodal architectures that explicitly model visual attention;
2. Design training objectives that encourage models to generate answers based on correct visual evidence;
3. Build more fine-grained evaluation metrics to quantify the causal relationship between visual evidence and answers.

Practical application insights: While pursuing accuracy, we need to equally emphasize interpretability and evidence tracing capabilities.