SciEvalKit: A Unified Framework and Leaderboard for Scientific Intelligence Evaluation

SciEvalKit is a scientific intelligence evaluation toolkit for large language models (LLMs) and multimodal models, covering the entire research workflow from literature review to experimental design, data analysis, and paper writing. It aims to address the limitation of traditional AI-in-science evaluation which is confined to single tasks, provide a standardized benchmark for evaluating the capabilities of AI in scientific research, and maintain an open leaderboard to track model performance.

With the increasing application of large language models (LLMs) and vision-language models (VLMs) in scientific research, traditional evaluation methods are limited to single tasks (such as question answering or summary generation), making it difficult to reflect the models' performance in real scientific research workflows. Scientific research is a multi-stage, multimodal continuous process, but most existing benchmarks only cover one or two stages, lacking systematic evaluation of end-to-end scientific research capabilities.

Developed by the InternScience team, SciEvalKit is an open-source evaluation toolkit that provides a unified and rigorous evaluation framework, including complete datasets, testing processes, and an open leaderboard. Its core feature is full workflow coverage: it breaks down the scientific research workflow into multiple key stages and designs specific evaluation tasks for each stage to comprehensively map the models' scientific research capabilities.

SciEvalKit's evaluation framework covers six core stages of the scientific research workflow:

1. Literature review and knowledge retrieval: Test the ability to locate, filter, and integrate information from massive literature;
2. Problem definition and hypothesis generation: Evaluate the ability to propose valuable research questions based on existing knowledge;
3. Experimental design and method selection: Assess the ability to design reasonable experimental plans and select appropriate research methods;
4. Data analysis and statistical inference: Test the ability to process experimental data, perform statistical analysis, and draw reliable conclusions;
5. Result interpretation and discussion: Evaluate the ability to explain research findings and discuss their significance and limitations;
6. Paper writing and academic communication: Test the ability to generate research papers that comply with academic norms.

SciEvalKit adopts a multi-level evaluation strategy:

• Objective question evaluation: Factual questions and method selection questions are automatically scored by matching with standard answers;
• Generative task evaluation: Open-ended tasks (such as paper writing) are evaluated using model-based automatic assessment (e.g., GPT-4) combined with expert manual review;
• Multimodal support: Tasks such as chart understanding and experimental image analysis are designed for VLMs;
• Domain coverage: Covers multiple disciplines including physics, chemistry, biology, medicine, and computer science; In addition, standardized evaluation scripts and interfaces are provided to facilitate researchers to integrate their models for testing.

The open leaderboard maintained by SciEvalKit provides a reference benchmark for the scientific research community:

• Objectively compare the differences in scientific research capabilities between different models;
• Identify the shortcomings of models and directions for improvement;
• Track the development trends and progress of model capabilities;
• Provide data support for model selection and application scenario matching; This system helps avoid leaderboard manipulation and over-promotion, and accurately presents the real capabilities of models.

SciEvalKit fills the gap in the field of AI-in-science evaluation: it provides developers with optimization goals and a fair competitive environment; it helps end-users identify models with scientific research assistance capabilities; it promotes the standardization and scientificization of evaluation methodologies in the field. As AI evolves into a scientific research partner, systematic evaluation becomes increasingly important, and SciEvalKit's full-workflow framework lays the foundation for future development.