Hybrid Reinforcement Learning and LLM-Based Agent Decision-Making Framework: Dual-Track Exploration in Wumpus World

This article introduces a Wumpus World solving framework integrating pure reinforcement learning and language model enhancement methods, exploring the implementation principles and comparative value of two technical routes: PPO-based recurrent neural networks and SFT+GRPO-based LLM reasoning and decision-making. Through its dual-track parallel design, this project provides a comparative sample for understanding the advantages and disadvantages of different AI paradigms.

Wumpus World is a classic test scenario in AI education. Agents need to navigate a partially observable grid world, avoid traps and the Wumpus monster, and find gold—testing their reasoning, risk assessment, and long-term planning abilities. A recent open-source project adopts a dual-track parallel design to compare pure reinforcement learning and LLM-enhanced technical routes, aiming to explore the characteristics of different AI paradigms.

The pure RL solution uses the PPO algorithm (a stable policy gradient method), limiting the magnitude of policy updates to avoid training oscillations. The key design is a Recurrent Neural Network (RNN) architecture, which uses its memory capacity to integrate historical observations and gradually build an internal cognitive map of the environment—simulating the behavior of human explorers piecing together a danger map, such as inferring safe paths from local clues.

The LLM route uses two-stage training: Supervised Fine-Tuning (SFT) to learn Wumpus World rules and decision-making patterns, followed by GRPO to optimize fine-tuning preferences. The core is to use the LLM as a reasoning engine, which receives natural language environment descriptions and generates reasoning processes and decisions (e.g., inferring the Wumpus's position based on perceptual clues). It has the advantages of strong interpretability and low sample dependency.

In terms of sample efficiency: the LLM method requires fewer interaction samples due to its pre-trained logical reasoning foundation. In terms of generalization ability: pure RL policies are tailored to specific environments and need retraining when the environment changes, while the LLM's general reasoning ability may adapt more easily to changes. In terms of interpretability: the LLM's explicit reasoning process is clear, whereas pure RL agents are 'black boxes'.

This framework provides researchers with a standardized testing platform and demonstrates the advantages of technical integration to practitioners. Future directions include hybrid agents (LLM for high-level planning, RL for low-level actions), transfer learning (LLM adapting to different grid world variants), etc., to help build general AI agents.

Wumpus World touches on core AI challenges (decision-making under uncertainty, exploration-exploitation balance, perception integration). This project provides a new perspective by comparing the two paradigms. Regardless of the merits of the methods, the comparative research attitude of 'letting data speak' is worthy of recognition, and maintaining an open curiosity about different paradigms is a key path to approaching general AI.