SpecVQA: A Benchmark for Scientific Spectrum Understanding and Visual Question Answering

SpecVQA is a professional scientific image benchmark aimed at evaluating the scientific spectrum understanding capabilities of multimodal large models. This benchmark covers 7 representative spectrum types (e.g., UV-Vis, infrared spectra, etc.), includes 620 carefully selected images and 3100 expert-annotated question-answer pairs, all data sourced from peer-reviewed scientific literature to ensure professionalism and quality.

Spectral images are common yet highly challenging data forms in scientific research, widely used in physics, chemistry, astronomy, and other fields. Their difficulties lie in unstructuredness and domain specificity: they contain professional information such as complex numerical relationships and peak features, requiring deep domain knowledge for interpretation. Existing multimodal models perform well in general visual tasks but struggle with professional spectral images.

Design of SpecVQA

• Test Scope: Covers 7 spectrum types (UV-Vis, IR, NMR, MS, XRD, Raman, Fluorescence), 620 images + 3100 expert question-answer pairs.
• Dual Evaluation Objectives: Scientific spectrum question-answer evaluation (information extraction, domain reasoning) and underlying task evaluation (peak identification, numerical reading, etc.).

Data Construction and Annotation

• Source: Peer-reviewed scientific literature to ensure authenticity and professionalism.
• Annotation: Completed by domain experts to guarantee the scientific nature of questions and accuracy of answers.
• Task Types: Direct information extraction (e.g., peak wavelength) and domain reasoning (e.g., compound structure judgment).

Spectral Data Processing

To address the token explosion and high computational cost issues of high-resolution spectra, an intelligent sampling (high density in key regions) + interpolation reconstruction strategy is adopted. This preserves key features while compressing data, and its effectiveness is verified through ablation experiments.

The research team tested multiple mainstream MLLMs and established a public leaderboard, finding:

• Information extraction outperforms reasoning: Models perform well in numerical reading and peak identification but struggle with domain reasoning tasks.
• Domain gap exists for general models: General models without scientific training have difficulty understanding the professional meaning of spectra.
• Large differences across spectrum types: Common types (e.g., UV-Vis) show better performance, while professional types (e.g., NMR) perform poorly.

Current models have a significant gap compared to human experts, requiring improvements in domain adaptation, numerical reasoning, and professional image understanding capabilities.

The release of SpecVQA is of great significance:

• Promote the development of scientific AI: Provide a standardized evaluation platform to incentivize the development of models better at scientific data understanding, accelerating scientific discovery and automated analysis.
• Expand model boundaries: Prove the feasibility of extending vision-language models to the scientific domain; future scientific assistants need to interpret professional charts.
• Facilitate interdisciplinary collaboration: The collaboration model between AI researchers and domain scientists paves the way for AI applications in more scientific fields.

SpecVQA is an important step for multimodal AI to move towards the professional scientific domain. It not only provides evaluation standards but also reveals technical limitations and directions. As models' spectrum understanding capabilities improve, AI will play a greater role in scientific research—from assisting experimental analysis to accelerating discovery, from educational popularization to industrial quality inspection, benefiting many fields.