HiPER: A Hierarchical Reinforcement Learning Framework for Large Language Model Agents

HiPER is an innovative hierarchical reinforcement learning framework that addresses the challenges of sparse rewards and credit assignment in multi-turn interactive tasks by explicitly separating high-level planning and low-level execution, achieving state-of-the-art (SOTA) performance on the ALFWorld and WebShop benchmarks.

• Original Author/Maintainer: JonP07 (Jiangweizhi Peng) and collaborators
• Source Platform: GitHub
• Original Title: HiPER-agent
• Original Link: https://github.com/JonP07/HiPER-agent
• Paper Link: https://arxiv.org/abs/2602.16165
• Source Publication Time: February 2026
• Conference: ICML 2026

---

Large Language Models (LLMs) acting as agents face severe challenges when performing multi-turn decision-making tasks in interactive environments. Especially in long-range tasks, reward signals are often sparse and delayed—agents may need to perform dozens or even hundreds of actions before receiving meaningful feedback.

Traditional reinforcement learning methods usually model LLM agents as flat policies with a single time scale, selecting one action per step. This design has fundamental flaws in sparse reward scenarios: credit must be propagated across the entire trajectory, lacking explicit temporal abstraction, leading to unstable optimization processes and inefficient credit assignment.

HiPER (Hierarchical Plan-Execute Reinforcement Learning) proposes an innovative hierarchical framework, whose core insight is: explicitly separating high-level planning and low-level execution.

This framework decomposes the policy into two collaborative components:

1. High-level Planner: Responsible for proposing subgoals and decomposing complex tasks into manageable sequences of subtasks
2. Low-level Executor: Responsible for converting each subgoal into a specific sequence of actions and executing them

This hierarchical architecture draws on the human intuition for problem-solving—we don't directly think about every muscle movement; instead, we first make a plan and then execute it step by step.

HiPER's core technical contribution is Hierarchical Advantage Estimation (HAE). This is the key to solving the credit assignment problem in hierarchical reinforcement learning.

Traditional Generalized Advantage Estimation (GAE) performs well in flat policies but faces challenges in hierarchical settings:

• Updates to high-level planning need to consider the cumulative effects of low-level execution
• Updates to low-level execution need to align with high-level goals
• The optimization objectives of the two levels need to be coordinated and unified

HAE addresses the above issues through the following mechanisms:

1. Execution-level Credit Assignment: Aggregate rewards for the execution process of each subgoal to evaluate the quality of that subgoal
2. Planning-level Credit Assignment: Evaluate the high-level planning strategy based on the completion status of subgoals
3. Cross-level Coordination: Ensure the update directions of the two levels are consistent and provide unbiased gradient estimates

Theoretical analysis shows that HAE has smaller variance compared to flat GAE, which means more stable training and faster convergence.

HiPER was evaluated on two challenging interactive benchmarks: