AI_Go_LLM: Testing Large Language Models' Spatial Reasoning and Decision-Making Capabilities Using Go

AI_Go_LLM is an open-source project that systematically evaluates large language models (LLMs) in complex spatial reasoning and strategic decision-making through the classic strategy game Go. The project reveals the strengths and limitations of current LLMs in symbolic reasoning tasks, providing a unique perspective for understanding the decision-making mechanisms of LLMs.

Large language models have achieved remarkable results in natural language processing tasks, but can they handle complex strategic tasks requiring precise spatial reasoning? Go poses unique challenges to LLMs: understanding 2D spatial relationships, evaluating long-term strategic value, and effectively searching through a vast state space. Unlike specialized Go AIs, LLMs lack explicit tree search mechanisms and Go-optimized architectures, but they possess extensive knowledge and pattern recognition capabilities. The core question of the project: Can these general capabilities compensate for the absence of specialized architectures?

AI_Go_LLM has built a complete evaluation framework that supports multiple mainstream LLMs to play against each other. The core is a text encoding system for converting board states, allowing LLMs to "understand" the Go board. A comparative study is conducted using coordinate representation (for precise calculation) and regional description (closer to human understanding). The evaluation system is divided into three layers: basic tests (rule understanding such as legal moves), intermediate tests (local tactics such as life-and-death judgment), and advanced tests (global strategic decision-making).

Experiments show that LLMs have excellent pattern recognition capabilities at the local tactical level, being able to handle common board patterns and joseki; however, their deep reading (multi-step variation prediction) is subpar, weaker than specialized Go engines, reflecting the limitations of the Transformer architecture in precise sequence reasoning. Additionally, LLMs exhibit systematic biases in judging territory ownership and calculating points, which may arise from training data distribution or limitations in numerical precision.

Through chain-of-thought analysis, LLM decisions exhibit an "intuitive" characteristic: quickly identifying candidate moves but struggling to conduct in-depth subsequent analysis, contrasting with the systematic search of specialized AIs. External prompts (tactical themes/strategic directions) can significantly improve performance, indicating that the models possess Go knowledge but lack the ability to independently organize and apply it.

Comparison tests show: Top-tier Go AIs (such as KataGo) are comprehensively ahead of LLMs; mid-tier open-source engines are on par with the strongest LLMs; LLMs may outperform the overall level in specific tactical problems. The strengths of LLMs lie in comprehensive judgment and creative moves (derived from extensive knowledge analogy), but specialized AIs achieve precise modeling and efficient search through Monte Carlo Tree Search (MCTS) + Convolutional Neural Networks (CNNs), making them more efficient in specific tasks.

The project's results have broad application value: spatial reasoning is the foundation of fields such as robot navigation and molecular design. The boundary of LLM capabilities provides a reference for hybrid architecture design (combining LLM general knowledge with specialized model precise calculation). Future research directions include: more efficient board encoding, hybrid architectures combining LLMs with lightweight search, multi-modal input testing, exploration of larger models, etc.