BeTTER Benchmark: Debunking the Illusion of Embodied Reasoning Capabilities in VLA Models

The BeTTER benchmark decouples high-level reasoning failures from low-level execution constraints for the first time using causal intervention and kinematic isolation methods, revealing severe cognitive deficits in semantic understanding and sequence planning in current VLA models. This thread will introduce core content such as background, methodology, and diagnostic findings in separate floors.

In recent years, Vision-Language-Action (VLA) models have achieved impressive success rates in robot manipulation benchmarks, demonstrating seemingly strong semantic understanding and sequence planning capabilities. However, teams from Peking University, Tsinghua University, and BeingBeyond question whether these successes mask deep cognitive deficits, and have launched the BeTTER benchmark to debunk the "illusion" of such capabilities.

Current evaluations conflate task completion with correct reasoning. Models may complete tasks through behavioral inertia (repeating high-frequency actions from training) rather than semantic understanding, or recognize objects but misunderstand their functional/spatial relationships. BeTTER refers to this as "embodied reasoning illusion", where traditional metrics only focus on results and ignore the cognitive process.

The core innovations of BeTTER are causal intervention and kinematic isolation:

• Causal intervention: Modify environmental variables (e.g., physical properties while keeping object appearance unchanged) to observe the model's sensitivity to semantically relevant interventions;
• Kinematic isolation: Decouple action outputs from a perfect executor to distinguish between cognitive failure (not knowing what to do) and execution failure (being unable to do it).

BeTTER evaluations reveal two major flaws in state-of-the-art (SOTA) VLA models:

1. Behavioral inertia: Over-reliance on specific action sequences leads to failure in generalized scenarios due to inability to adapt flexibly;
2. Semantic feature collapse: Recognize visual features of objects but fail to establish mappings to functional attributes (e.g., knowing a cup but not understanding its use as a container).

BeTTER includes 10 basic manipulation tasks + 60 diagnostic variants, manipulating object properties, spatial configurations, etc., to form a multi-dimensional evaluation grid. It also provides data augmentation, privileged logging tools, integrates with MimicGen to generate training data, and supports analysis of internal model representations.

BeTTER adopts a progressive open-source strategy, having already released papers and frameworks, with plans to open task generation pipelines and more in the future. Dependent on tools like Objaverse and MimicGen, it calls for the establishment of an evaluation system that better reflects real cognitive capabilities to promote the maturity of embodied intelligence technology.