Strategy-Induct: A Task-Level Instruction Induction Framework Without Annotated Answers

This article introduces the Strategy-Induct framework, whose core innovation is the ability to induce task-level instructions without annotated answers. The framework forms (strategy, question) pairs by generating explicit reasoning strategies, outperforms existing SOTA methods in the pure question setting, and finds that combining LLMs with reasoning models further improves performance. It addresses the bottleneck of traditional instruction induction relying on annotated data, and has application values such as lowering the threshold for prompt design and enhancing model interpretability.

Problem Background: Annotation Bottleneck in Instruction Generation

In LLM applications, high-quality prompt design is crucial, but manual design is time-consuming and relies on expert experience. Existing instruction induction methods depend on input-output pair examples and require annotated answers. However, obtaining annotated data is difficult in real-world scenarios (e.g., open-ended Q&A, complex reasoning tasks), which limits their scope of application.

Core Idea of Strategy-Induct

The core innovation of Strategy-Induct is to completely break away from dependence on annotated answers, allowing it to induce effective instructions with only example questions. The framework has two stages:

1. Generate explicit reasoning strategies to form (strategy, question) pairs;
2. Induce task instructions from strategy-question pairs.

The intuition behind this is: describing "how to think" (strategy) is easier to infer and more generalizable than "correct answers". As an intermediate representation, strategies retain the core features of the task and avoid noise and bias from answers.

Detailed Technical Approach

Strategy Generation Stage

Construct prompt templates to guide the model to generate abstract and actionable reasoning strategies for each example question (e.g., math problem strategy: Identify type → Extract values → Establish equations → Solve and verify).

Instruction Induction Stage

Extract commonalities from strategy-question pairs and generate natural language instructions that describe the essence of the task and the reasoning framework.

Application During Inference

Prepend the induced instructions as system prompts to guide the model to reuse task-specific reasoning patterns.

Experimental Design and Key Results

The experiments cover scenarios such as mathematical reasoning, commonsense reasoning, and code generation, using a pure question protocol (only questions are provided):

• Outperforms SOTA: Performs better than existing answer-free instruction induction methods on multiple benchmarks;
• Cross-model Consistency: The advantage remains consistent across models of different scales (billions to hundreds of billions of parameters);
• Collaborative Improvement: Combining LLM-generated strategies and instructions with specialized reasoning models for execution yields better performance than using a single model.

Application Value and Practical Significance

1. Lower Threshold: Optimized instructions can be obtained with just example questions, facilitating rapid prototyping and vertical domain applications;
2. Enhanced Interpretability: Explicit strategies allow users to understand the model's reasoning process, making it easier to debug and optimize (suitable for high-risk scenarios like healthcare and law);
3. Strategy Intervention Space: Adjustments can be made at the strategy level (merging, prioritization, domain-specific patterns) without modifying the model.

Limitations and Future Directions

Limitations:

• Mainly targeted at single-turn reasoning tasks; needs improvement for complex multi-turn/tool-use tasks;
• Performance is greatly affected by the quality of example questions.

Future Directions:

• Explore hierarchical strategy representations to support complex reasoning;
• Develop active learning mechanisms to select valuable examples;
• Optimize the collaboration between strategy induction and model fine-tuning.