Tractatus-Eval: An Evaluation Benchmark for Spatial Embodied Logical Capabilities of Large Language Models

Tractatus-Eval is an evaluation benchmark for the spatial embodied logical capabilities of large language models, inspired by Wittgenstein's philosophy. It aims to quantify the capability boundaries of LLMs in spatial embodied reasoning tasks and reveal the cognitive limitations of text-only models. Through six physical reasoning tasks and a zero-contamination verification mechanism, this benchmark provides a reliable measurement tool for the AI research community, helping to understand the capability boundaries of LLMs and guide the design of next-generation systems.

The project name is derived from Wittgenstein's assertion in Tractatus Logico-Philosophicus: 'The limits of my language mean the limits of my world'. The core question is to explore the cognitive limits of constructing a world through text alone. Using a systematic evaluation method, the project quantifies the performance of LLMs in embodied physical reasoning tasks and reveals the fundamental gap between text-only models and the cognition of the real physical world.

Six Evaluation Tasks

1. Spatial Navigation and Path Planning: Tests obstacle impassability, boundary constraints, and path coherence
2. Key-Lock Puzzles and State Tracking: Requires tracking inventory states and action sequence dependencies
3. Object Stacking and Structural Stability: Tests understanding of gravity and support constraints
4. Container Water Filling and Volume Conservation: Tests capacity limits and overflow handling
5. Collision Prediction and Trajectory Tracking: Tests time extrapolation and trajectory simulation capabilities
6. Circuit Connectivity and Switch Logic: Tests topological connectivity and Boolean logic

Zero-Contamination Data Generation

Through a physics engine replay validator, the execution process of distractors is simulated, and only those that violate physical constraints are retained, ensuring a zero-contamination rate for the benchmark.

1. Scale Does Not Equal Capability: The Pythia family shows increased parameters but accuracy lower than the random baseline (25%)
2. Training Data Is More Critical: The 2.7B-parameter Phi-2 outperforms 7B Mistral and 8B Llama-3, benefiting from code and math-intensive training data
3. Task Difficulty Stratification:
   • Difficult Tasks: Spatial Navigation, Key-Lock Puzzles (Phi-2 accuracy: 32-33%)
   • Partially Solvable: Object Stacking, Container Water Filling (Phi-2:40-67%)
   • Unsolvable: Collision Prediction, Circuit Connectivity (all models at ~50% random level)

Empirical verification of Wittgenstein's insight: Text-only models do not interact with the physical world; their understanding of concepts like 'impassable' and 'gravity' remains at the symbolic level, and they cannot acquire true embodied cognition.

For physical reasoning scenarios, text-only models are insufficient; external validators, deterministic rule engines, or multimodal perception capabilities need to be introduced. The gap can be bridged through preference alignment (DPO) and external guardrails (e.g., NeMo Guardrails).

Tractatus-Eval is a rigorously designed evaluation benchmark. Through a systematic approach, it reveals the fundamental limitations of LLMs in embodied spatial reasoning, provides a reliable measurement tool for AI research, and points to the direction for next-generation AI system design.