EVENT5Ws: A Large-Scale Dataset and Benchmark Study for Open-Domain Document-Level Event Extraction

EVENT5Ws is a large-scale, manually annotated, and statistically validated open-domain event extraction dataset. It aims to address the limitations of existing datasets, such as limited coverage of event types and lack of large-scale manually verified data, providing a new benchmark for training generalized event extraction algorithms. This article will introduce it from aspects including background, dataset characteristics, methodology and workflow, experimental evaluation, etc.

Event extraction is a core task in natural language processing, crucial for event understanding, situational analysis, and emergency decision support. Existing event extraction datasets have two major limitations: most are confined to closed domains with limited coverage of event types; open-domain scenarios lack large-scale, manually verified high-quality datasets, which restricts the development of general-purpose algorithms.

EVENT5Ws is a large-scale manually annotated dataset designed specifically for open-domain document-level event extraction. Its core features include:

• Substantial scale: Provides sufficient training samples to support deep learning model training
• Manual fine annotation: All annotations are completed by professionals and statistically validated
• Open-domain coverage: Not limited to specific domains, covering diverse event types
• Systematic workflow: Clear methodological support for all links from data collection to quality control

The construction of EVENT5Ws follows a systematic methodology: designing detailed annotation specifications (defining event concepts, element classification, and boundary determination clearly); adopting a multi-round review mechanism (initial annotation, cross-checking, expert sampling) to ensure consistency. The dataset focuses on extracting the 5W event elements: Who (participants), What (event type and action), When (time), Where (location), Why (cause and background), with structured representation facilitating downstream applications.

The research team used EVENT5Ws to evaluate mainstream pre-trained language models and established the first performance benchmark:

1. Existing models still have room for improvement in handling complex open-domain document-level events
2. Data scale brings significant benefits; trained models show good learning ability and generalization potential
3. Strong cross-regional generalization ability, which can effectively adapt to datasets in different geographical contexts

The release value of EVENT5Ws:

• For researchers: Provides a standardized evaluation platform, promotes technological progress, and supports exploration of new model architectures
• For application developers: Models can be used in scenarios such as news analysis, public opinion monitoring, and intelligence analysis
• For dataset builders: Summarizes experience in large-scale dataset development, which can be transferred to other NLP tasks

Limitations of EVENT5Ws: It mainly focuses on English text and lacks annotations for event timing and causal relationships. Future directions: Expand to multilingual versions; integrate other event knowledge bases to build a more comprehensive system; combine in-context learning of large language models to explore few-shot adaptation to new event types.

EVENT5Ws fills the gap of lacking large-scale manually verified datasets in the field of open-domain event extraction. Through systematic annotation workflow, strict quality control, and comprehensive benchmark evaluation, it provides a solid foundation for the research and development of event extraction algorithms. It shows good performance in cross-regional generalization experiments, and the model has strong practical value, making it an important resource for practitioners and researchers in fields such as information extraction and knowledge graph construction.