SciDef: A Research Toolkit for Automatically Extracting Definitions from Academic Literature Using Large Language Models

SciDef is an open-source research project aimed at using Large Language Models (LLMs) to automatically extract term definitions from massive academic literature, addressing the pain point of low efficiency in manual definition lookup under academic information overload. The project provides a complete pipeline, evaluation scripts, and two high-quality manually annotated datasets (DefExtra for definition extraction and DefSim for definition similarity), laying a reproducible resource foundation for academic literature understanding and knowledge extraction research.

The number of academic publications is growing exponentially, with thousands of new papers released daily, and researchers are facing the challenge of information overload. Understanding the precise definitions of professional terms is a basic requirement for academic research, but manual lookup and organization of definitions are time-consuming and prone to omissions, while traditional keyword search or manual reading is inefficient. The SciDef project addresses this core issue by proposing an LLM-based automated solution to intelligently extract definitions and evaluate similarity, while providing open-source resources to support subsequent research.

The core components of SciDef include an LLM-driven definition extraction pipeline that uses the DSPy framework to optimize prompts (supporting open-source/proprietary models); Technically, it is developed in Python, uses uv for package and environment management, has a clear structure (including scripts for pipeline, artifacts for prompt templates, and docs for guidelines), and supports CLI operations. The application of DSPy reduces the cost of prompt tuning and makes the reproduction of different models more convenient.

The project provides two core datasets:

1. DefExtra: Used for definition extraction evaluation, containing 268 definitions from 75 papers (60 on media bias, 15 non-related), with annotation markers supporting PDF reconstruction;
2. DefSim: Used for similarity evaluation, containing 60 pairs of definitions and manual similarity scores from 1 to 5. The evaluation framework covers multi-model comparison, prompt strategy analysis, similarity metric calculation, and NLI benchmark testing to ensure the validation of the solution's effectiveness.

SciDef has a wide range of application scenarios:

• Automated literature review: Quickly collect domain term definitions to accelerate review writing;
• Knowledge graph construction: Extract definitions as nodes to support semantic relationship modeling;
• Educational tool development: Generate glossaries to assist learning and writing;
• Interdisciplinary research: Help understand terms from other fields and lower communication barriers.

When using SciDef, note the following:

• The public version of DefExtra only provides marker positions; users need to reconstruct the complete text from their own PDF copies;
• Some documents are AI-assisted generated; it is recommended to verify commands and configurations;
• The repository history is compressed, so commit records do not reflect the actual distribution of author contributions. These statements reflect the team's emphasis on academic integrity.

SciDef is an important advancement in the field of academic literature processing. As a complete research infrastructure (data, code, evaluation, documentation), it paves the way for subsequent research. It provides a valuable starting point for researchers in NLP, information retrieval, or scientific knowledge mining. The project has released datasets on Hugging Face (mediabiasgroup/DefExtra, DefSim); welcome to use and improve them.In the future, with the improvement of LLM capabilities, extensions based on SciDef are expected to push the boundaries of automatic understanding of academic literature.