Knowledge Graph-Enhanced Vision-Language Models: A New Approach to Improving Physical World Reasoning Capabilities

This project (VLM-Reasoning-Model-using-Knowledge-Graph) was published by tirth1263 on GitHub (link: https://github.com/tirth1263/VLM-Reasoning-Model-using-Knowledge-Graph, release date: 2026-05-23). Its core idea is to enhance the physical world reasoning capabilities of vision-language models (VLMs) by combining knowledge graphs (KGs) with explicit physical rules. Compared to fine-tuning methods, this zero-shot reasoning enhancement strategy is lighter and more interpretable, and has achieved certain improvements on the ScienceQA physics validation set.

Vision-language models (VLMs) perform well in tasks such as image understanding and visual question answering, but they have limitations when dealing with physical common sense reasoning problems (e.g., shadows and lighting, buoyancy and density, heat conduction, etc.). Traditional VLMs lack explicit physical knowledge representation and rely on statistical patterns in training data to guess, making it difficult to understand physical causal laws.

The project adopts a neuro-symbolic hybrid approach, combining external knowledge graphs (such as ConceptNet) with VLMs. The core steps include: 1. Object grounding (identifying physical objects in the problem); 2. Knowledge retrieval (obtaining relevant physical facts from KGs); 3. Semantic filtering (screening relevant knowledge); 4. Rule triggering (applying handwritten physical rules like shadow and buoyancy rules); 5. Constructing KG-enhanced prompts; 6. Generating answers and comparing; 7. Ablation experiments to verify component contributions. Compared to LoRA fine-tuning, this zero-shot method avoids template memorization issues and has better generalization.

Evaluation on the ScienceQA physics validation set (121 questions) shows: PaliGemma-3B baseline accuracy is 28.1%; using only ConceptNet KG increases it to 30.6%; KG + physical rules further increases it to 31.4%. Ablation experiments indicate that random knowledge harms performance, verifying the importance of knowledge quality; LoRA fine-tuning has poor generalization due to template memorization.

Project insights: Explicit knowledge injection during reasoning may be more effective than implicit learning during training (physical common sense's structured features are suitable for symbolic representation); neuro-symbolic hybrid architecture can complement shortcomings; interpretable reasoning processes are suitable for educational scenarios (helping students understand physical principles); the framework can be extended to chemistry, biology, and other fields.

Current limitations: Handwritten rules have limited coverage (complex scenarios need expansion); high cost of manually writing rules; verified only on small models; increased reasoning latency. Future directions: Automatically extract physical rules; verify gains for large models; optimize retrieval efficiency to reduce latency.

This project demonstrates the feasibility of using knowledge graphs + explicit rules to enhance VLM physical reasoning. The zero-shot reasoning enhancement method is lightweight, interpretable, and easy to iterate, providing a practical case for neuro-symbolic hybrid AI systems. Although the improvement is limited, with the maturity of knowledge tools, this method is expected to be applied in more fields.