Reading
The MultiSmolVLA project combines the 4M-21 multi-modal encoder with SmolVLA and introduces a modality dropout training strategy, which significantly enhances the robustness of vision-language-action (VLA) models in sensor failure scenarios, providing a more reliable perception solution for robot applications.
Section 01
EPFL's MultiSmolVLA project addresses the fragility of single-RGB VLA models in real-world robot scenarios. It combines the 4M-21 multi-modal encoder with SmolVLA and introduces a modality dropout training strategy to boost robustness against sensor failures, aiming to provide more reliable perception solutions for robot applications.
Section 02
Current VLA models like π0 and OpenVLA rely on RGB input but face performance drops in real scenarios: sensor failures (hardware issues), environmental interference (glare, smoke), and occlusions. These lead to catastrophic failures, posing safety risks for robots.
Section 03
Architecture: Replaces SmolVLA's SigLIP encoder with Apple's 4M-21 multi-modal encoder, which fuses RGB, depth, semantic segmentation, and thermal modalities into a unified token sequence. Training: Uses a progressive modality dropout curriculum—zero dropout in connector alignment phase, then linear increase to 0.5 in robustness fine-tuning—to teach the model to adapt to missing modalities.
Section 04
Thermal Data: Uses ThermalGen (diffusion model) to synthesize thermal images from RGB, converted via ImageBind to 4M-21-compatible embeddings. Two-Stage Training: 1) Train MLP connector (4M-21 → SmolLM2 space) with no dropout; 2) LoRA fine-tune SmolLM2 and action expert with increasing dropout. Dataset & Eval: Uses LIBERO benchmark (4 task categories: Spatial, Object, Goal, Long) with 3 test conditions: clean, hard dropout, soft corruption.
Section 05
Baseline results: Vanilla SmolVLA (87.3% avg task completion), Vanilla π0 (86%). Full MultiSmolVLA performance not disclosed, but ablation studies validate: 1) Impact of additional modalities vs RGB-only; 2) Effectiveness of curriculum dropout vs fixed dropout.
Section 06
Contributions: 1) Shifts focus to 'performance under failure' for VLA models; 2) Demonstrates assembly-style innovation (combining existing components);3) Curricula dropout can be applied to medical imaging, autonomous driving;4) Open-sourced code/evaluation for community.
Section 07
Limitations: Synthetic thermal data may differ from real; higher computation cost vs single RGB; no discussion on sensor synchronization. Future: Explore efficient fusion (cross-attention), adaptive modality selection, extend robustness to adversarial attacks/calibration errors.
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
Cross-generational hardware testing based on NVIDIA A10G and H100 GPUs, comparing and analyzing the throughput, latency, and concurrent scalability of three mainstream LLM inference frameworks. SGLang achieves a 3.4x performance advantage over vLLM on H100, while Ollama faces architectural bottlenecks in high-concurrency scenarios.