OpenAlex Abstract Quality Crisis: One-Eighth Have Completeness Issues

A systematic evaluation found that 12% of English journal paper abstracts in the widely used OpenAlex database have completeness issues, with insufficient content and metadata misalignment being the main defects. These issues have far-reaching impacts on computational science research based on literature data (such as knowledge graph construction, automatic literature review, etc.). This article will analyze from the aspects of background, methods, findings, impacts, root causes, and response strategies.

In the fields of computational science and bibliometrics, paper abstracts have become key research data, supporting applications like knowledge graph construction, impact assessment, and automatic reviews. The premise is reliable data quality. As an open academic database, OpenAlex integrates over 250 million academic works and metadata, making it the preferred data source for computational science research. However, there was a lack of systematic evaluation of its abstract completeness before.

The research team used a sample of 10,000 English journal paper abstracts, evaluated through two-stage annotation (expert manual review + LLM-assisted classification), and defined 7 completeness failure modes: insufficient content, metadata misalignment, non-abstract content, duplicate content, formatting errors, language issues, and other issues.

Evaluation results show that 12% of abstracts have issues. Among the failure mode distribution, insufficient content (35%) and metadata misalignment (30%) account for the highest proportions: insufficient content manifests as overly short abstracts, repeated titles, etc.; metadata misalignment manifests as author information or keywords mixed into abstracts, directly affecting downstream text analysis tasks.

Problematic abstracts can lead to distorted topic clustering in knowledge graphs, contamination of training data for automatic literature reviews, misjudgment of research hotspots in scientific research policy formulation, etc., posing threats to the reliability and accuracy of computational science research.

Root causes include: data source heterogeneity (inconsistent metadata standards across different sources), limitations of automated processing (PDF parsing errors, field mapping errors), and resource constraints in quality control (limited scale of manual review).

Researchers should implement data cleaning, sample validation, sensitivity analysis, and transparent reporting; the community can crowdsource annotations and feedback issues through collaboration platforms; technically, solutions like LLM-assisted detection, multi-source verification, and publisher cooperation can be adopted.

The OpenAlex case reveals the quality paradox of open data: the tension between openness and quality control. In the future, it is necessary to establish quality evaluation standards, benchmark datasets, and human-machine collaborative quality assurance mechanisms to jointly solve the quality problems of open academic data. Key points: 12% of abstracts have issues, main defects are insufficient content and metadata misalignment, requiring joint response from researchers, community, and technology.