Innovative Application Research of Open-Source Large Language Models in Software Metadata Entity Disambiguation

This article introduces a study that uses open-source large language models to solve the problem of software metadata entity disambiguation. Key contents include: constructing a multi-annotator benchmark dataset, comparing three reasoning strategies (direct prompting, self-consistency, multi-step agent-based), exploring feasible paths for high-precision entity resolution in noisy and heterogeneous data environments, and providing academic institutions and enterprises with new reproducible and controllable data governance ideas.

In the scientific research software ecosystem, metadata has quality issues such as naming ambiguity, version confusion, and inconsistent descriptions, making entity disambiguation tasks extremely challenging. Traditional rule-matching methods are ineffective against heterogeneous noisy data, and commercial API solutions are costly and have privacy concerns. Therefore, the EvaMart team explores local deployment of open-source large language models to achieve reliable entity disambiguation.

Task Definition: Formalize entity disambiguation as a ternary classification problem (same software/different software/insufficient evidence). The input contains multi-modal evidence (name, description, webpage, code repository, etc.), and the output is a structured result with confidence and reasoning basis.

Dataset: Construct approximately 1000 real cases (from OpenEBench), adopt a multi-annotator mechanism and calculate Cohen's Kappa to ensure quality, and also set up a balanced subset to handle class imbalance.

The study compares three strategies:

1. Direct Prompting: Input all information at once, lowest cost but unstable performance in complex cases;
2. Self-Consistency: Majority voting after multiple sampling inferences, improves reliability but increases computational cost;
3. Agent-Based Multi-Step: Simulate human reasoning process (evidence extraction → diagnosis → targeted retrieval → decision → verification), strong ability to handle complex cases but with the most calls (5-6 times), and triggers targeted retrieval instead of guessing when evidence is insufficient.

Experiment: Deploy open-source models locally on HPC with no commercial API dependencies; record all parameters through configuration files, and generate manifest.json after running to save environment information to ensure reproducibility.

Engineering: The data directory is immutable; model outputs are stored in an independent runs directory; prompt versions are managed via file names; the code is environment-independent and can be tested locally or deployed on HPC.

Results: Direct prompting has the lowest cost but limited accuracy; self-consistency exchanges moderate cost for performance improvement; the agent-based strategy has significant advantages in complex cases; an uncertainty-aware scheme (automatic decision for high confidence, manual review otherwise) is proposed to balance quality and efficiency.

Significance: Prove that open-source models can achieve commercial API-level effects in specific tasks while maintaining data sovereignty; provide a technical framework and benchmark dataset for the construction of scientific research software knowledge graphs.

Outlook: Explore more efficient reasoning strategies, expand task types, and refine uncertainty quantification methods.