AutoVLA: An End-to-End Autonomous Driving Vision-Language-Action Model Driven by Adaptive Reasoning and Reinforcement Fine-Tuning

AutoVLA, a NeurIPS 2025 work proposed by UCLA Mobility Lab, aims to build a safer and more intelligent end-to-end autonomous driving system through unified vision-language-action modeling, adaptive reasoning mechanism, and reinforcement learning fine-tuning. The project is open-sourced on GitHub, with a release date of May 29, 2026.

Traditional end-to-end autonomous driving with modular design has problems of information transmission loss and error accumulation; although Visual-Language Models (VLM) have strong scene understanding capabilities, their application in autonomous driving faces three major challenges: real-time performance, safety, and long-tail scenarios. AutoVLA is thus proposed to solve inter-module problems through unified modeling, and address VLM application difficulties by combining adaptive reasoning and reinforcement learning.

1. Unified Vision-Language-Action Architecture: Integrates perception, reasoning, and action modules to achieve end-to-end optimization, enhance interpretability, and transfer pre-trained knowledge; 2. Adaptive Reasoning Mechanism: Dynamically adjusts reasoning depth based on scene complexity (shallow for simple scenes, deep for complex/critical scenes) to balance efficiency and decision quality; 3. Reinforcement Fine-Tuning (RFT): Designs a comprehensive reward function (safety, comfort, efficiency) and optimizes strategies by combining PPO algorithm and human feedback.

• Multimodal Input: Processes surround-view images (6 cameras), vehicle status, navigation information, and historical trajectories; uses ViT as the visual encoder to support high resolution; - Linguistic Scene Description: Converts visual features into structured language (e.g., scene, surrounding vehicles, pedestrians, and suggested actions) to improve interpretability; - Action Generation: Adopts a hybrid action space (discrete decision + continuous control) to balance interpretability and precision.

Evaluated on nuScenes, Waymo, and CARLA simulation datasets, the results outperform baselines: planning accuracy L2 error reduced by 27% (0.85→0.62m), collision rate reduced by 67% (0.12%→0.04%), comfort score increased by 18% (7.2→8.5), and inference latency reduced by 21% (120→95ms). Ablation experiments validate: removing adaptive reasoning increases latency by 40%/reduces performance in complex scenes by 15%; removing RFT increases collision rate by 0.05%/reduces comfort by 0.7; single-view input reduces planning accuracy by 0.16m.

• Computational Efficiency Optimization: INT8 quantization (volume reduced by 75%/speed increased by 2x), knowledge distillation (small models maintain performance), dynamic batching; - Safety Redundancy: Rule-based fallback (covers model decisions in critical scenes), uncertainty quantification (triggers takeover when confidence is low), continuous monitoring (automatic degradation on anomalies).

Current Limitations: Simulation-to-reality gap, performance in extreme weather needs improvement, insufficient data for long-tail scenarios, high peak computing demand; Future Directions: Integrate world models (long-term planning), multi-vehicle collaboration, continuous learning (adapt to new scenarios), neuro-symbolic fusion (reliability in extreme scenarios).

Core contributions of AutoVLA: Unified architecture simplifies design, adaptive computing balances efficiency and performance, reinforcement learning surpasses human strategies, and language representation enhances interpretability. Insights: Autonomous driving requires targeted innovations (architecture/reasoning/training) rather than blindly pursuing large models, helping end-to-end technology move from research to application.