CTRL: A Continual Test-Time Reinforcement Learning Framework for Large Language Models

CTRL (Continual Test-Time Reinforcement Learning) is a continual test-time reinforcement learning framework for large language models, specifically addressing two core challenges in online adaptation of reasoning task streams: error accumulation and catastrophic forgetting. It integrates techniques like process reward model-guided trajectory selection, posterior correction, output-process distillation, cognitive anchor replay, and conflict-aware gradient projection, effectively improving the stability of continual learning and reasoning capabilities. Experiments verify that its performance outperforms existing methods.

Although large language models (LLMs) acquire massive knowledge during pre-training, it is difficult to obtain optimal answers for complex reasoning tasks with a single forward pass. Test-Time Reinforcement Learning (TTRL) enables 'learning while thinking' through additional computational optimization during the reasoning phase, but online adaptation in continuous task streams faces two major issues:

1. Error Accumulation: Relying on majority voting pseudo-labels to guide training leads to cumulative and amplified errors, resulting in performance degradation;
2. Catastrophic Forgetting: Gradient updates from new tasks overwrite effective reasoning patterns of old tasks, causing the model to forget solutions to early problems.

The coupling of these two issues makes designing a robust continual learning framework extremely challenging.

CTRL is a complete engineering framework whose core design philosophy is to optimize current task performance while protecting learned knowledge. It includes five key technical components:

1. Process Reward Model-Guided Trajectory Selection: Uses fine-grained rewards for intermediate steps to filter high-quality candidate trajectories, which is more reliable than majority voting;
2. Posterior Correction Mechanism: Dynamically adjusts the confidence of pseudo-labels based on Bayesian posterior inference to reduce the impact of noise;
3. Output-Process Distillation: Distills both the final answer and reasoning process to learn rich strategies instead of just memorizing answers;
4. Cognitive Anchor Replay: Maintains a buffer of anchor samples for key knowledge points, mixing them during training to stabilize old knowledge;
5. Conflict-Aware Gradient Projection: Analyzes the directional relationship of task gradients and uses projection adjustments to mitigate conflicts between new and old tasks.

CTRL was tested on mathematical reasoning benchmarks such as AMC-TTT, AIME-TTT, and MATH-TTT, covering models like Qwen3 and the Llama series. Comparisons with methods like TTRL and INTUITOR show:

• Accuracy Improvement: The final average accuracy is significantly higher than that of comparison methods;
• Reduced Forgetting: The forgetting metric is close to zero, effectively preserving old knowledge.

These results verify the synergistic effect of each component and the feasibility of continual test-time reinforcement learning.

CTRL is implemented based on the open-source reinforcement learning library verl. Key modules include:

• cttrl_local_prm.py: Local PRM client
• cttrl_memory.py: Cognitive replay buffer management
• cttrl_prm_client.py: API PRM client
• cttrl_utils.py: Trajectory selection utility functions
• ppo_trainer_cttrl.yaml: Training configuration

Users can modify the configuration to adapt to task types, base models, etc. It supports multi-GPU training (optimized for 8 GPUs by default).

Insights from CTRL: A well-designed combination of mechanisms can achieve effective continual learning without real labels, making it suitable for scenarios with high annotation costs. Future directions include:

1. Extending to tasks such as code generation and multimodal reasoning;
2. Exploring more efficient anchor selection strategies;
3. Combining model editing techniques to achieve fine-grained knowledge updates.

Developers can refer to the CTRL implementation to build continual learning capabilities.