PhysicsFormer: A Causal Reasoning Framework for Language Models to Truly Understand the Physical World

On June 7, 2026, the UWM research team open-sourced PhysicsFormer on GitHub—a lightweight physical reasoning model with only 82 million parameters. By encoding physical scenes into structured state tensors, this model achieved an accuracy of 79.6% on the CLEVRER physical reasoning benchmark, outperforming large-scale language models like Llama-3.3-70B, which proves the critical role of physics-based representations in causal reasoning. Original project link: https://github.com/uwm-se/PhysicsFormer.

Current large language models (LLMs) perform well on text tasks, but they have limitations when handling causal reasoning in the physical world—often relying on statistical pattern matching rather than true physical understanding. The CLEVRER benchmark requires models to understand object interactions, predict future states, and perform counterfactual reasoning. These tasks are highly challenging for pure language models lacking physical grounding, exposing their limitations.

The core of PhysicsFormer is to explicitly encode physical scenes into structured state tensors: each object is represented by a 35-dimensional vector (including attributes like position, velocity, mass, material, color, shape, etc.), combined into a [1,N,35] tensor. The architecture includes: a physics encoder (FullPhysicsFormer, which extracts visual-physical features), a base language model (a lightweight variant of DistilGPT-2), an adapter (PhysicsLLMAdapterV2, which connects the two via prefix tuning + LoRA), and auxiliary heads (handling numerical regression, classification, and multiple-choice tasks).

A three-stage progressive training strategy is adopted:

1. Stage 1: Freeze the language model, train the adapter's MLP layers and auxiliary heads using losses like generative cross-entropy and numerical MSE, with a learning rate of 2e-4;
2. Stage 2: Add LoRA to DistilGPT-2's attention layers (405,000 additional parameters), introduce InfoNCE contrastive loss to prevent representation collapse, with a learning rate of 5e-5;
3. Stage 3: Fully fine-tune all parameters of DistilGPT-2, keep the objective functions from the first two stages, with a learning rate of 2e-5. This strategy avoids the optimization difficulties of direct end-to-end training.

The experimental results are significant:

• Overall accuracy on CLEVRER validation set: 79.6% (explanatory: 78.9%, predictive: 76.4%, counterfactual: 81.5%);
• 3-6 object held-out partition: PhysicsFormer 69.2% vs Llama-3.3-70B's 62.5% (statistically significant);
• 15-object stress test: 64.6% on predictive questions, far exceeding DeepSeek-V3 (53.8%) and Llama-3.3-70B (48.8%);
• Ablation experiment: Accuracy dropped from 82.3% to 6.9% after zeroing out physical state tensors, proving dependence on physical representations;
• ComPhy zero-shot test: Demonstrates cross-benchmark transfer capability.

Technical Insights:

1. Structured representation is more important than model size (82M parameters outperform 70B parameter models);
2. New idea for multimodal fusion: Convert vision to physical structured representation first, then reason;
3. Progressive training is effective (unlock parameters in stages);
4. Open-source and reproducible (provides code, pre-trained checkpoints, and reproduction guidelines). Future directions: Handle more complex scenes, expand coverage of physical attributes, balance specialization and generality.

Limitations:

1. Scene complexity constraints (trained on 3-6 object scenes; complex real-world scenes need verification);
2. Limited coverage of physical attributes (does not involve phenomena like fluids, deformation, electromagnetism, etc.);
3. Trade-off between specialization and generality (optimized for physical reasoning; need to explore methods to maintain generality).

PhysicsFormer represents an important progress in the field of AI physical reasoning, proving that small models can outperform large general-purpose models through physics-based representations. Its physical grounding approach provides a new direction for multimodal AI design, and also paves the way for connecting perception, reasoning, and action in embodied intelligence and robotics, promoting the construction of intelligent systems that truly understand the physical world.