UniEdit: A Unified Knowledge Editing Evaluation Benchmark for Large Language Models

UniEdit is a large-scale open-domain knowledge editing evaluation benchmark accepted by NeurIPS 2025. It contains 311,000 samples, built from 29.9 million entities in Wikidata, covering 25 subject areas. Using the NMCS algorithm to generate diverse evaluation samples, it systematically assesses the performance of knowledge editing algorithms across three dimensions: reliability, generalization, and locality.

Large Language Models (LLMs) solidify a large amount of factual knowledge during training, but this knowledge may be outdated, incorrect, or contain biases. Knowledge editing technology aims to precisely modify specific facts in the model without retraining the entire model. However, existing evaluation benchmarks have obvious limitations: narrow knowledge coverage, insufficient structural diversity, and incomplete evaluation criteria, making it difficult for researchers to fully understand the real performance of editing algorithms.

UniEdit was developed by researchers, containing 311,000 high-quality samples, built by sampling from 29.9 million entities in Wikidata, covering 25 specific disciplines across five major fields. Its core innovation is the NMCS (Neighborhood Multi-hop Chain Sampling) algorithm, which can generate complex test cases such as multi-hop reasoning chains, same-entity reasoning paths, and relation reversal scenarios based on factual triples. The research team also used the DeepSeek-V3 model to automatically convert structured data into natural language form, improving data construction efficiency and annotation quality.

UniEdit covers 25 fields including agriculture, art, astronomy, biology, chemistry, computer science, etc.; supports single-hop and multi-hop reasoning chain structures; and includes diverse evaluation scenarios such as relation reversal, subject-object aliases, and specificity tests. Each editing sample is equipped with complete entity descriptions, relational attributes, and reasoning path annotations, providing rich information for in-depth analysis.

UniEdit evaluates knowledge editing algorithms from three key dimensions:

1. Reliability: Measures the accuracy of the model's memory of the target fact after editing;
2. Generalization: Assesses the model's ability to generalize to semantic variants of the edited fact and reasoning tasks (such as paraphrasing, multi-hop reasoning, etc.);
3. Locality: Tests the precision of the editing operation to ensure that irrelevant facts are not affected.

For knowledge editing researchers, UniEdit provides a standardized and reproducible evaluation environment, helping to fairly compare the pros and cons of different algorithms; for LLM developers, it can help identify model knowledge defects and verify repair effects; in practical applications, knowledge editing technology can correct misinformation, update outdated knowledge, eliminate harmful biases, and improve the credibility and safety of AI systems.

UniEdit pushes knowledge editing evaluation to a new height through large-scale data construction, innovative sampling algorithms, and a comprehensive evaluation framework. As LLMs are widely applied in key fields, precise and efficient knowledge editing capabilities are becoming increasingly important. UniEdit provides a solid foundation for this research direction and is expected to accelerate the practicalization of knowledge editing technology.