In-depth Evaluation of OpenAI o1 Model's Planning Capabilities: Analysis of Feasibility, Optimality, and Generalization

The VITA research team at the University of Texas at Austin presented a study at the NeurIPS'24 LanGame workshop, systematically evaluating the feasibility, optimality, and generalization of GPT-4 and the o1 series models (o1-mini, o1-preview) in planning tasks. The study reveals: the o1 models excel in problem understanding, being able to parse complex domain definitions more accurately; however, they have obvious limitations in spatial reasoning (executing errors during multi-step reasoning) and generalization (performance degradation when symbolic representations change). This research provides empirical references for the application and subsequent research of LLM planning capabilities.

With the rapid development of large language models, AI planning capabilities have become a focus in academia and industry. The OpenAI o1 series has attracted attention for its strong reasoning ability, but its performance in complex planning tasks remains to be verified. The goal of this study is to evaluate the o1 models in three key dimensions of planning tasks: feasibility, optimality, and generalization. The test benchmarks selected classic planning domains (such as the Barman bartender problem and the TyreWorld tire replacement problem), covering different complexities to test structured reasoning abilities.

The research team conducted parallel comparative tests on GPT-4, o1-mini, and o1-preview. Experimental process: Convert PDDL-formatted problem descriptions into natural language prompts, and observe the models' ability to generate solutions. A multi-difficulty test set was constructed, with each case containing complete domain definitions and problem instances, requiring understanding of constraints and generating executable action sequences. Randomized symbolic encoding variants were introduced for testing to evaluate the models' robustness to problem representation forms, and to determine whether they truly understand the internal structure rather than relying on pattern matching.

Advantages: The o1 series significantly outperforms GPT-4 in problem understanding, being able to parse complex domain definitions more accurately and identify key state variables and action preconditions, indicating improvements in its reasoning mechanism for structured information processing.

Limitations: In spatial reasoning, it is prone to 'correct thinking but wrong execution' during multi-step reasoning (understanding the goal but having logical gaps or constraint violations in the action sequence); in generalization, performance degradation is more than expected when random symbols replace original vocabulary, indicating that the model relies on specific patterns in training data rather than abstract essence.

1. Establish verification mechanisms: AI systems relying on planning capabilities need to use PDDL solvers for formal verification of model-generated plans to ensure correctness.
2. Adopt hybrid architectures: Use o1 for high-level intent understanding and initial plan generation, and specialized planning algorithms for detailed verification to leverage their respective advantages.
3. Optimize prompting methods: The MEMO work (context optimization to improve planning capabilities) can enhance performance without modifying the model.

1. Improve spatial reasoning capabilities: Add structured geometric and topological information to training data.
2. Enhance robustness of symbolic reasoning: Better handle changes in representation forms.
3. Develop more effective evaluation benchmarks: Add complex real-world test scenarios (such as the MindGames competition).
4. Combine neural and symbolic reasoning: Explore the organic combination of neural network intuitive reasoning and traditional symbolic AI precise reasoning, which requires architectural innovation.

This study provides empirical data for understanding the real capabilities of the o1 model. Although o1's reasoning ability has improved compared to previous generations, its planning capabilities still have significant limitations. It reminds practitioners to remain sober when evaluating LLMs—both recognize progress and face shortcomings squarely. It emphasizes the importance of benchmark testing: through strict and comprehensive evaluation, understand the boundary of model capabilities and make reasonable technical selections. We look forward to substantial breakthroughs in AI planning capabilities in the future.