SceneTeract: Bridging the Gap Between Vision-Language Models and 3D Scene Understanding via Physical Verification

This article introduces the SceneTeract framework, which aims to evaluate the functional affordances of 3D scenes by combining high-level semantic reasoning with low-level geometric verification. The study finds that current state-of-the-art vision-language models (VLMs) have systematic biases in judging physical feasibility, and proposes using SceneTeract as a reward engine for post-training VLMs. The core of SceneTeract is a grounded verification engine that supports agent-specific functional assessment. Evaluations on synthetic scenes and VLMs reveal key flaws in existing systems, providing a technical foundation for the physical grounding of embodied AI.

The vision of Embodied AI is to enable AI systems to perceive, understand, and act like humans, relying on the ability to understand what a 3D scene "can do"—known as functional affordance in cognitive science. Assessing functional affordances faces four major challenges:

1. Separation of semantics and geometry: Knowing "this is a chair" does not mean knowing "can I sit on it";
2. Agent dependency: The same object has different affordances for different agents (child/adult/wheelchair user);
3. Compositional complexity: Complex activities require combinations of multiple atomic action sequences;
4. Physical constraints: Geometric factors such as reachability, clearance, and navigability determine functional feasibility.

The core innovation of SceneTeract is the grounded verification engine, which combines high-level semantic reasoning with low-level geometric checks. Its workflow has three stages:

1. Activity decomposition: Split complex activities (e.g., "making breakfast") into atomic action sequences;
2. Constraint verification: Verify agent-specific accessibility (reachability, clearance, navigability) for each atomic action;
3. Physical simulation: Use explicit physical and geometric simulations to verify constraint satisfaction. The framework introduces agent profiles (physical parameters, operational capabilities, constraints) to enable agent-specific functional assessment.

The research team used SceneTeract to evaluate popular synthetic 3D indoor scene datasets, automatically generating activity queries (e.g., "taking a book from the shelf") and verifying their feasibility. Key findings: Synthetic environments have frequent functional failures, including:

• Unreachable objects: Placed in positions that agents cannot reach;
• Blocked pathways: Furniture layouts block navigation paths;
• Incompatible sizes: Chair height does not match table height;
• Logical contradictions: Drawers are blocked by other objects and cannot be opened. These failures are not visually obvious but cause problems when "used", which is a blind spot for pure visual methods.

When evaluating current state-of-the-art VLMs, they were shown 3D scenes and candidate activities to judge feasibility, which was compared with SceneTeract's physical verification. It was found that VLMs have systematic mismatches between semantic confidence and physical feasibility:

1. Overconfidence: Giving high-confidence feasible judgments for obviously infeasible activities (e.g., "sitting on a floating chair");
2. Scale blindness: Difficulty perceiving relative object sizes (e.g., a child reaching an adult-sized shelf);
3. Lack of physical intuition: Missing concepts like gravity and support (e.g., "placing in the air" without considering support);
4. Agent generalization failure: Same reachability judgment for wheelchair users and walkers. This reveals that VLMs learn visual-language correlations rather than physical causality.

The study proposes using SceneTeract as a reward engine for VLM post-training, with the following workflow:

1. Data generation: Automatically generate large-scale scene-activity-feasibility labeled data;
2. Reward modeling: Physical verification results serve as reward signals (positive for feasible, negative for infeasible);
3. Policy optimization: Fine-tune VLMs using reinforcement learning or preference optimization to internalize geometric constraints. Advantages include: scalability (automatic data generation), accuracy (physical simulation as ground truth), flexibility (customizable training), and interpretability (rewards correspond to explicit physical constraints).

The implementation of SceneTeract's verification engine includes:

• Geometric primitives: Reachability cones (calculating reachable areas), navigation meshes (walkable paths), clearance detection (operation space), collision detection (action collisions);
• Physical simulation: Rigid body dynamics (object movement), joint constraints (moving parts like doors/drawers), friction and contact (grasp stability);
• Activity representation: Hierarchical structure (activity = atomic action sequence; atomic action = operation type + target object + constraints).

Current Limitations:

1. Scene coverage: Mainly evaluates synthetic indoor scenes; real-world complexity is higher;
2. Activity scope: Focuses on daily home operations; professional scenes need expansion;
3. Simulation accuracy: Physical simulation differs from the real world; material properties are hard to model precisely;
4. Computational cost: High overhead for geometric verification, making real-time operation difficult.

Future Directions:

• Real-world deployment: Integrate with robot platforms to handle perceptual noise;
• Learning acceleration: Neural approximation methods for fast feasibility predictors;
• Multimodal expansion: Integrate touch/audio to support human-machine collaboration;
• Social dimension: Consider social norms for multi-agent social scenarios.