CrashChat: A Multimodal Large Language Model for Traffic Accident Video Analysis

CrashChat is a multimodal large language model specifically designed for traffic accident video analysis, improved based on the VideoLLaMA3 architecture. It supports six core tasks including accident recognition, time localization, causal reasoning, and prevention recommendation generation. The project has built an instruction fine-tuning dataset containing 18,385 videos and 96,184 question-answer pairs. It has been accepted by the ICPR 2026 conference, and the code, model weights, and dataset have been open-sourced. It has application potential in multiple scenarios such as intelligent traffic monitoring and insurance claims settlement.

With the development of intelligent transportation and autonomous driving, traffic accident analysis has become a key direction. Traditional manual review of surveillance videos is inefficient and makes it difficult to extract patterns. Existing general-purpose multimodal large language models lack specificity for the traffic accident domain, making it hard to handle both visual perception tasks (vehicle and pedestrian recognition) and advanced cognitive tasks (causal reasoning, liability determination) simultaneously, and they cannot accurately understand the dynamic process and underlying causes of accidents.

CrashChat uses VideoLLaMA3-7B as its backbone and adopts the LoRA fine-tuning strategy to reduce training costs. The team explored three multi-task training strategies: independent single-task model (baseline), homogeneous multi-task model (grouped by language/perception), and heterogeneous multi-task model (unifying all tasks). Experiments show that the heterogeneous strategy, while maintaining simplicity, achieves performance comparable to or even better than the single-task model.

The training data comes from real-scenario datasets such as MM-AU and Nexar. After video extraction and annotation, question-answer pair generation, and quality screening, a dataset containing original and scaled versions was built (already open-sourced). The evaluation covers dimensions such as accuracy and time localization precision. Results show that CrashChat significantly outperforms general video understanding models in metrics like accident recognition accuracy and causal reasoning rationality.

CrashChat can be applied in:

1. Intelligent traffic monitoring: Real-time accident recognition and triggering emergency responses;
2. Insurance claims assistance: Assisting in understanding accident processes and liability attribution;
3. Driving training and education: Generating accident cause analysis and prevention recommendations;
4. Autonomous driving research and development: Providing accident scenario benchmark testing and capability evaluation.

CrashChat has the following improvement directions:

1. Multi-view fusion: Extending to multi-camera collaborative analysis;
2. Extreme weather scenarios: Improving performance under low visibility conditions such as rain, fog, and night;
3. Real-time inference optimization: Developing lightweight deployment solutions for edge devices;
4. Cross-domain generalization: Enhancing adaptability to traffic scenarios in different countries/regions.

CrashChat is fully open-sourced: The paper was published on arXiv (arXiv:2512.18878) and accepted by ICPR 2026; the code is hosted on GitHub; model weights and datasets are uploaded to Hugging Face. The deployment environment is based on Python 3.10 and PyTorch 2.4, supports CUDA 11.8, depends on FlashAttention, FFmpeg, etc., and the scripts support single/multi-GPU configurations.