VHG: Validator-Enhanced Hard Problem Generation Framework, Breaking the Bottleneck of LLM Training Data

Core Viewpoint: VHG (Validator-Enhanced Hard Problem Generation Framework) constructs a tripartite self-play mechanism by introducing an independent validator, decoupling problem validity assessment from difficulty assessment, and solving the bottleneck where LLMs struggle to generate valid, challenging, and novel problems. It significantly outperforms existing baselines in indefinite integral and mathematical reasoning tasks, providing a high-quality solution for LLM training data expansion, autonomous scientific research, etc.

The Ceiling of LLM Capabilities: Problem Generation Dilemma

Large language models perform well in solving scientific and mathematical problems, but generating valid, challenging, and novel problems is a long-standing bottleneck.

Importance of Problem Generation

• Training data expansion: Breaking the bottleneck of quality and diversity in LLM training data
• Capability boundary exploration: Systematically detecting the weak points of models
• Autonomous scientific research: AI needs to propose valuable questions rather than just answer them
• Educational applications: Generating personalized practice questions

Dilemma of Existing Methods

• Dependence on human experts: High quality but high cost and difficult to scale
• Traditional self-play trap: Binary framework (problem setter-solver) easily leads to reward hacking (generating invalid/trivial problems)

VHG's New Tripartite Self-Play Paradigm

Problems with Traditional Binary Framework

The problem setter's goal is to make the solver fail, which easily leads to invalid/trivial/memory-dependent problems.

Core of Tripartite Framework: Introducing Validator

• Problem setter: Generates candidate problems
• Solver: Evaluates difficulty
• Validator: Independently verifies validity (decouples validity and difficulty)

Joint Reward Mechanism

Problem setter's reward = validity score (assessed by validator) + difficulty score (assessed by solver), eliminating reward hacking.

Two Validator Variants

• Hard symbolic validator: Based on CAS (e.g., SymPy), rigorous and deterministic, suitable for formal solution domains (indefinite integrals, etc.)
• Soft LLM validator: Flexible and widely applicable, uses prompts to let LLMs verify, suitable for open-ended reasoning

Experimental Evaluation Results

Indefinite Integral Task

• Validity improvement: Invalid problems (non-integrable functions) are almost eliminated
• Difficulty control: Covers from basic to advanced techniques
• Diversity: Covers multiple integration techniques such as substitution and integration by parts

General Mathematical Reasoning Task

• Quality: Higher manual evaluation scores, more educational/research value
• Novelty: Generates variants not present in training data, avoiding overfitting
• Solvability: All problems are verified to be solvable

Key Principles of VHG's Effectiveness

The Power of Decoupling

• Goal separation: Validity first, then difficulty, avoiding sacrificing validity
• Independent optimization: Validator and solver apply different pressures, exploring a richer problem space
• Composability: Validator and solver can be improved independently

Essence of Adversarial Training

The tripartite triangular relationship (problem setter vs solver/validator) is more stable and less prone to mode collapse

Curriculum Learning Potential

• Progressive difficulty: Guides generation of sequences from simple to difficult
• Capability matching: Personalized problem generation
• Continuous challenge: Generates harder problems as the solver's ability improves

VHG Application Scenarios

LLM Training Data Enhancement

• Continuously generates novel problems, avoiding data exhaustion
• Dynamically adjusts difficulty, generating targeted data for weak areas

Intelligent Education Platform

• Personalized practice question generation
• Targeted intensive training (based on error patterns)
• Dynamically adjusts difficulty to maintain optimal learning state

Benchmark Construction

• Generates high-quality leak-free test problems
• Ensures training/test set isolation
• Covers different capability dimensions

Autonomous Scientific Research

• Automatically generates hypotheses and experimental designs
• Explores new proof paths for mathematical conjectures
• Discovers potential connections between domains

Limitations and Future Directions

Current Limitations

• Validator construction cost: Hard validators require expert knowledge, soft validators are not strict enough
• Domain specificity: Currently focused on mathematics; expanding to physics etc. requires significant work
• Creativity limitation: Dependence on manual judgment for problem creativity/research value
• Computational overhead: Tripartite framework requires more resources

Future Directions

• General validator: Cross-domain framework reduces expansion cost
• Multi-objective optimization: Introduce goals like educational value and research significance
• Human-machine collaboration: Expert guidance + VHG generation
• Meta-learning: Quickly build domain validators
• Theoretical analysis: Research on convergence properties of tripartite games

VHG provides an effective solution for high-quality mathematical problem generation through its tripartite self-play framework, outperforming existing methods in experiments and being scalable to broader scientific fields. As LLM capabilities improve, high-quality training data generation becomes a key bottleneck, and VHG will play an important role in AI training and autonomous scientific research.