Breaking the RLVR Capability Ceiling: A Latent Variable Markov World Model Based on Variational Autoencoders

This paper proposes a latent variable Markov world model based on Variational Autoencoders(VAE) to address the structural issue of non-Markov state representation in Reinforcement Learning Post-training(RLVR).By learning compact latent state representations of reasoning trajectories to replace full token history and introducing an uncertainty-driven exploration mechanism, this model achieves a paradigm shift from "sampling efficiency improvement"to "capability boundary expansion",providing a new path breaking RLVR capability ceiling.

Methods like RLVR and GRPO are mainstream for enhancing large models'reasoning capabilities,but they have structural flaws:the state received by policy networks is an unbounded,redundant full token history(non-Markov).This causes RLVR to only improve sampling efficiency within existing capability ranges,failing to break the reasoning ceiling(only redistributing solution path probabilities without discovering new strategies).Yue et al.theoretically proved this problem in 2025;Yuan&Xie’s 2026 study introduced Markov states but still operated in token space.

The core of this method is end-to-end learning of structured latent states in solution space,including three pillars: 1.Latent State Representation:VAE trained on reasoning trajectories;encoder maps trajectory hidden states to latent distribution(μ,σ²),decoder reconstructs trajectories; 2.Uncertainty Exploration:Use VAE posterior variance as signal(high variance→explore,low→exploit),KL divergence as intrinsic reward term; 3.Policy Conditioning:RL policy receives VAE-sampled latent variable z instead of raw token history,enabling Markov state operations.

The experiment selected MATH-B-I subset from MATH-Beyond benchmark(difficult problems with base model pass@1024=0) and set four control groups(sharing base model,reward,decoding budget):

• baseline_grpo Full token history(standard RLVR);
• token_markov_grpo Yuan method reproduction(token-space Markov predictor);
• latent_grpo VAE latent state(with no uncertainty reward);
• latent_grpo_uncertainty Complete method(VAE latent state+KL exploration reward).

The project uses modular design: VAE State Encoder:Input backbone model hidden state sequence;2-3 layer MLP outputs latent distribution;latent dimension 64-128;training objective ELBO; Uncertainty Reward Module:Intrinsic reward β_t×KL(q(z|τ)||p(z)),β annealed during training; Training Framework:Policy backbone Qwen2.5-1.5B-Instruct;RL algorithm GRPO via TRL;hyperparameters learning rate 1e-6,KL coefficient 0.001,batch size128,group size8.

This work extends world model philosophy from physical environments to abstract reasoning(e.g.LeWM predicts physical frames,this model models cognitive states.Future directions:

• Replace VAE with diffusion models for richer belief state denoising;
• Add world model dynamics to support multi-step reasoning planningin latent space;
• Extend applications to code generation,theorem proving,scientific discovery fields.

The latent variable Markov world model points out that token-level history is not an appropriate state representationfor reasoning.Breaking RLVR ceiling requires learning structured latent representationsin solution spaces.By explicitly incorporating cognitive uncertainty exploration,this method expects to achieve paradigm shift from "sampling efficiency improvement"to "capability boundary expansion",providing new directions for further enhancing large models'reasoning capabilities.