CoReason: An Evaluation Framework for Generative AI Reasoning Capabilities and a Collaborative Reasoning Model

The CoReason project proposes a systematic collaborative reasoning competency evaluation framework, with a three-dimensional competency model (framework construction, judgment evaluation, guidance and control) at its core, aiming to evaluate and enhance the collaborative reasoning capabilities between humans and generative AI. The project includes resources such as papers, evaluation tools, and datasets, which are open-source and have educational and practical value.

The rapid development of generative AI (e.g., GPT, Claude) has made human-AI collaborative reasoning a reality, but there is a lack of a systematic framework for effectively evaluating and enhancing the ability of humans and AI to reason together. Traditional AI evaluation focuses on model performance and ignores cognitive abilities and strategy application in human collaboration, which led to the proposal of the CoReason project.

Collaborative reasoning refers to humans and generative AI completing complex reasoning tasks through interactive dialogue, emphasizing bidirectional cognitive participation. The three-dimensional competency model includes:

• Framework construction: Defining problem boundaries, setting goals, organizing background information, etc.;
• Judgment evaluation: Critically examining AI outputs, such as fact-checking and logical verification;
• Guidance and control: Dynamically adjusting interaction strategies, such as asking clarifying questions and correcting paths.

CoReason provides a complete evaluation toolkit:

• Standardized evaluation scale: A structured questionnaire based on the three-dimensional model to quantify competency levels;
• Contextualized test tasks: Reasoning tasks close to real scenarios to observe actual performance;
• Behavior coding framework: A systematic recording method to analyze key behavior patterns in human-AI dialogue.

The project is open-source under the MIT License, including: evaluation tool code, pre-trained model compatibility tests, benchmark datasets and annotation guidelines, and experimental design and analysis reports. The open and transparent approach facilitates the reproduction, verification, and expansion of the framework by academia and industry.

The value of the CoReason framework includes:

• AI literacy cultivation: Helping students shift from passive use to active collaboration;
• Professional training: Improving the efficiency and quality of knowledge workers' collaboration with AI;
• Product design guidance: Providing AI developers with reference to user competency models to optimize interaction interfaces.

CoReason marks a shift in human-AI collaboration research from technology-centric to competency-centric, focusing on how humans can better collaborate with AI. Future directions include cross-cultural adaptability research, domain-specific customized frameworks, real-time feedback training systems, and long-term collaborative competency tracking. This project provides a scientific framework for understanding and enhancing human-AI collaboration, and is an important foundational work for digital literacy education.