SafeVL: A Driving Safety Assessment System Based on Fine Reasoning of Vision-Language Models

SafeVL leverages the fine reasoning capabilities of Vision-Language Models (VLMs) to provide a comprehensive safety assessment solution for autonomous driving scenarios. It addresses the limitations of traditional methods (rule-based, pure visual, end-to-end deep learning) by offering reliable, explainable safety judgments that can identify potential hazards and their sources. Key features include multi-modal understanding, structured reasoning, and human-interpretable outputs, which are critical for building trust in autonomous driving systems.

Autonomous driving safety is a key bottleneck for large-scale deployment. Traditional methods have notable limitations:

• Rule-based: Relies on predefined rules, fails to cover complex/unexpected scenarios.
• Pure visual: Dependent on large labeled data, lacks explainability.
• End-to-end deep learning: Black-box nature makes safety verification and troubleshooting difficult. VLMs offer new possibilities with multi-modal understanding and reasoning, which SafeVL applies to driving safety assessment.

SafeVL uses VLMs for its core technology due to:

1. Multi-modal understanding: Handles both visual (camera images) and text (queries) info, mimicking human perception.
2. Reasoning & explainability: Shows step-by-step reasoning instead of direct outputs, crucial for safety-critical applications.
3. Generalization: Pre-trained VLMs adapt well to unseen scenarios, reducing reliance on specific labeled data.

Its fine reasoning framework includes:

• Scene decomposition: Breaks down complex scenes into sub-scenarios (road conditions, surrounding vehicles, pedestrians, environment).
• Multi-dimensional assessment: Evaluates space (relative positions), time (future trajectories), causality (event chains), and compliance (traffic rules).
• Progressive reasoning: Uses Chain-of-Thought to identify objects → analyze states → assess interactions → detect conflicts → judge safety level → suggest actions.

SafeVL's architecture consists of four layers:

1. Data input: Integrates multi-view cameras (360° perception), vehicle state data (speed, acceleration), and map/navigation info.
2. Visual encoder: Supports CLIP-style (general), SAM-style (segmentation), or dedicated driving encoders (domain-adapted).
3. Reasoning engine: Includes query generation (auto or external), multi-round reasoning controller (adaptive depth), and knowledge retrieval (traffic rules/accident cases).
4. Output generation: Provides safety levels (safe/attention/warning/danger), risk localization (annotated images), reasoning explanations (natural language), and response suggestions (e.g., deceleration).

Dataset: Collected from real driving records, simulators, and public datasets (nuScenes, Waymo), with multi-level annotations (scene labels, object attributes, interactions, reasoning explanations). Training:

• Pre-training: On large general V-L data for basic multi-modal skills.
• Domain adaptation: On driving data to learn traffic domain knowledge.
• Reasoning reinforcement: Supervised fine-tuning with reasoning annotations + RL with human feedback. Evaluation metrics:
• Accuracy: Safety classification, risk detection, collision prediction.
• Explainability: Consistency with experts, readability of explanations, decision point localization.
• Practicality: Inference latency, false/missed alarm rates, alignment with human judgment.

SafeVL can be applied in:

1. ADAS: As an intelligent module for nuanced safety assessments (e.g., considering relative speed, brake lights, road curvature).
2. Autonomous driving validation: Independent tool to evaluate self-driving decisions and resolve disagreements between systems and humans.
3. Accident analysis: Reconstructs pre-accident scenes to identify key factors, aiding algorithm improvement, insurance claims, and liability determination.
4. Driver training: Real-time assessment of trainees' driving safety, providing objective feedback and correct practices.

Current limitations:

• High compute resource demand (challenging for on-board embedded devices).
• Limited coverage of extreme scenarios (rare weather, special roads).
• Inconsistent reasoning results due to generative nature of VLMs.

Future directions:

• Edge optimization: Lightweight VLMs for on-board chips via compression/quantization.
• Continuous learning: Online learning from real deployments to adapt to new scenarios/rules.
• Multi-vehicle collaboration: V2V communication for shared perception and beyond-single-vehicle safety assessment.

SafeVL demonstrates the potential of VLMs in driving safety assessment. Its fine reasoning framework provides accurate, explainable safety judgments, which are essential for building human trust in autonomous driving. As technology matures, SafeVL-like systems are expected to become standard tools for autonomous driving safety validation, promoting the industry toward safer development.