TTVS: Enhancing Self-Exploratory Reinforcement Learning via Test-Time Variational Synthesis

TTVS (Test-Time Variational Synthesis) is a new framework that allows large reasoning models to self-evolve during the testing phase without labeled data. Addressing the limitation of traditional reinforcement learning (e.g., RLVR) which relies on high-quality labeled data, it helps models learn the intrinsic logic of problems rather than surface text patterns by dynamically generating semantically equivalent query variants, ultimately achieving better performance than supervised reinforcement learning. Its core consists of two modules: online variational synthesis and test-time hybrid exploration.

Traditional reinforcement learning (e.g., RLVR) relies on a large number of verifiable labeled reward signals. However, in professional/emerging fields such as medical diagnosis and legal consultation, the cost of obtaining labeled data is extremely high or even impossible. Existing test-time adaptation methods are limited by static query sets, prone to overfitting text patterns, and their performance drops sharply when facing similar problems with different expressions.

Online Variational Synthesis

Convert static test queries into dynamic training streams, generate multiple semantically equivalent but differently expressed variants (e.g., adjust wording/order while keeping core logic unchanged), forcing the model to learn problem structures rather than text patterns to alleviate overfitting.

Test-Time Hybrid Exploration

Adopt a balanced strategy:

• Accuracy-driven exploitation: Prioritize variants that are likely to produce correct answers for in-depth reasoning;
• Consistency-driven exploration: Verify the consistency of reasoning across different variants to avoid local optima or resource waste.

Experiments on 8 model architectures show that TTVS is universal. Using only unlabeled test data, it not only outperforms other test-time adaptation methods but also surpasses the state-of-the-art supervised reinforcement learning techniques trained with large amounts of high-quality labeled data.

• Reduced data dependence: Significantly reduce the demand for expensive labeled data, breaking through bottlenecks in professional fields;
• Enhanced adaptability: Allow models to continue evolving after deployment, achieving "lifelong learning" to cope with real-world changes;
• Push the boundaries of self-supervision: Verify that self-supervised methods are more effective than supervised learning in specific scenarios, providing directions for future research.

Currently, TTVS still has room for improvement:

1. The quality of variational synthesis needs to be improved to ensure variants are both diverse and semantically equivalent;
2. Adaptive adjustment methods for balance parameters in the hybrid exploration strategy need to be studied. Future research can focus on these directions for in-depth optimization.