InteractComp: An Interactive Reasoning Evaluation Framework for Large Language Models

InteractComp is a professional evaluation framework for the interactive reasoning capabilities of large language models (LLMs), aiming to fill the gaps in existing evaluation systems. It shifts the evaluation perspective from the traditional static "one-question-one-answer" mode to a dynamic interaction process, focusing on key capabilities such as the model's questioning strategy, context consistency, and decision quality in multi-turn dialogues, helping developers systematically identify model shortcomings and make targeted improvements.

Traditional LLM evaluations mostly use a static "one-question-one-answer" mode, focusing only on the accuracy of the final answer. However, real-world tasks (such as customer service and scientific research collaboration) require models to gradually understand problems, collect information, and make decisions through multi-turn interactions. InteractComp was created to address this need, focusing on evaluating the model's interactive reasoning capabilities.

The framework consists of three core components: 1. Configurable interactive task environment (defines goals, action space, state transition rules); 2. Multi-dimensional evaluation metrics (task completion rate, interaction efficiency, information acquisition strategy, decision quality, context consistency); 3. Extensible task library (modular design, supports adding new tasks, with built-in tasks in domains like information retrieval and puzzle solving).

The framework can be applied in multiple scenarios: 1. Customer service simulation: Evaluate the model's questioning strategy, problem understanding, and appropriateness of solutions; 2. Research assistant: Test the model's ability to acquire professional knowledge and apply scientific research methodologies; 3. Interactive teaching: Evaluate the model's ability to adjust teaching strategies based on student feedback.

Technical highlights include: 1. Standardized interface: Unifies the interaction method between models and the environment, lowering the threshold for integrating new models; 2. Reproducible experiment management: Supports random seed control, configuration versioning, and result recording to ensure experimental rigor; 3. Visualization analysis tools: Provides interaction trajectory playback, decision tree graphical display, etc., to facilitate model behavior diagnosis.

The framework's value for model development is reflected in: 1. Identifying capability shortcomings: Precisely locates the model's deficiencies in interactive reasoning (e.g., weak decision-making ability, low questioning efficiency); 2. Guiding fine-tuning strategies: Builds targeted training data based on evaluation results and supports export to training formats; 3. Reference for model selection: Provides comparative evaluation of multiple models to help developers choose the model suitable for specific scenarios.

Current limitations: Limited size of the task library, insufficient realism of environment simulation, and subjectivity in quantifying some metrics (e.g., questioning quality). Future directions: Expand the complexity of task environments, support multi-agent interaction, integrate real user data, and develop automated improvement suggestion functions.