MultiSmolVLA：通过模态丢弃训练增强VLA模型的多传感器鲁棒性

EPFL's MultiSmolVLA project addresses the fragility of single-RGB VLA models in real-world robot scenarios. It combines the 4M-21 multi-modal encoder with SmolVLA and introduces a modality dropout training strategy to boost robustness against sensor failures, aiming to provide more reliable perception solutions for robot applications.

Current VLA models like π0 and OpenVLA rely on RGB input but face performance drops in real scenarios: sensor failures (hardware issues), environmental interference (glare, smoke), and occlusions. These lead to catastrophic failures, posing safety risks for robots.

Architecture: Replaces SmolVLA's SigLIP encoder with Apple's 4M-21 multi-modal encoder, which fuses RGB, depth, semantic segmentation, and thermal modalities into a unified token sequence. Training: Uses a progressive modality dropout curriculum—zero dropout in connector alignment phase, then linear increase to 0.5 in robustness fine-tuning—to teach the model to adapt to missing modalities.

Thermal Data: Uses ThermalGen (diffusion model) to synthesize thermal images from RGB, converted via ImageBind to 4M-21-compatible embeddings. Two-Stage Training: 1) Train MLP connector (4M-21 → SmolLM2 space) with no dropout; 2) LoRA fine-tune SmolLM2 and action expert with increasing dropout. Dataset & Eval: Uses LIBERO benchmark (4 task categories: Spatial, Object, Goal, Long) with 3 test conditions: clean, hard dropout, soft corruption.

Baseline results: Vanilla SmolVLA (87.3% avg task completion), Vanilla π0 (86%). Full MultiSmolVLA performance not disclosed, but ablation studies validate: 1) Impact of additional modalities vs RGB-only; 2) Effectiveness of curriculum dropout vs fixed dropout.

Contributions: 1) Shifts focus to 'performance under failure' for VLA models; 2) Demonstrates assembly-style innovation (combining existing components);3) Curricula dropout can be applied to medical imaging, autonomous driving;4) Open-sourced code/evaluation for community.

Limitations: Synthetic thermal data may differ from real; higher computation cost vs single RGB; no discussion on sensor synchronization. Future: Explore efficient fusion (cross-attention), adaptive modality selection, extend robustness to adversarial attacks/calibration errors.