PTR：基于知识图谱的政治时序推理语言模型评估框架

PTR Project Overview

PTR is an open-source evaluation framework that uses knowledge graph-driven methods to systematically assess large language models (LLMs) on political temporal reasoning tasks. It includes a complete dataset, evaluation tools, and experiment reproduction workflows.

Key Basic Information

• Author/Maintainer: iguillenp
• Source: GitHub (https://github.com/iguillenp/ptr)
• Release Time: 2026-05-25

This framework aims to fill the gap in evaluating LLMs' domain-specific reasoning abilities, especially in the under-researched area of political temporal reasoning.

With the rapid development of LLMs, evaluating their domain-specific reasoning capabilities has become increasingly important. Political temporal reasoning is a challenging but under-researched field—it requires models to understand both relationships between political entities and the dynamic evolution of these relationships over time.

Traditional LLM evaluations often focus on general knowledge QA or simple logical reasoning, lacking systematic methods for tasks that combine domain knowledge, time dimensions, and complex causal relationships. PTR was created to fill this gap.

PTR adopts an innovative knowledge graph-driven evaluation approach. Its core idea is to formalize political temporal reasoning tasks as query and reasoning problems on a knowledge graph.

The structured political knowledge graph includes:

• Nodes: Political entities (countries, leaders, political parties, policies, etc.)
• Edges: Time-varying relationships between entities

Advantages of this paradigm:

1. Strong interpretability: Transparent reasoning paths via knowledge graphs
2. Good scalability: Easy to expand new entities and relationship types
3. Precise temporal modeling: Accurate assessment of models' grasp of historical evolution via timestamps
4. Domain-specific: Designed for political field characteristics, avoiding limitations of general evaluation tasks

PTR's code repository includes key components forming a complete evaluation workflow:

Data Layer

A carefully constructed political temporal dataset covering:

• Entity Types: Political figures, government agencies, political parties, policy issues, geographic regions, etc.
• Relation Types: Affiliation, policy positions, time-series events, causal relationships, etc.
• Time Span: Data covering different historical periods, supporting cross-period reasoning evaluation

Query & Evaluation Module

The queries directory contains query templates for various reasoning tasks:

• Temporal prediction: Predict subsequent developments given historical event sequences
• Relation inference: Infer implicit temporal relationships between entities
• Conflict detection: Identify temporal contradictions in the knowledge graph
• Path reasoning: Multi-hop reasoning based on graph paths

Experiment Reproduction Tools

Scripts (experiments.sh) and Jupyter Notebooks (KGC.ipynb, TR.ipynb, Results.ipynb) are provided to facilitate experiment reproduction and extended research.

PTR designs a multi-dimensional evaluation metric system:

Accuracy Metrics

• Hit Rate: Proportion of correct answers from the model
• Mean Reciprocal Rank (MRR): Measures the quality of correct answer rankings
• Precision & Recall: For binary classification reasoning tasks

Temporal Sensitivity Metrics

• Time Order Correctness: Evaluate the model's understanding of event sequence
• Duration Estimation Error: Measure the accuracy of event duration prediction
• Temporal Consistency: Detect temporal logical contradictions in model outputs

Robustness Metrics

• Adversarial Sample Performance: Stability under perturbed inputs
• Out-of-Distribution Generalization: Adaptability to unseen political entities or periods

The PTR framework has important practical significance in multiple fields:

Academic Research

Provides a standardized LLM evaluation benchmark for political science and computational social science researchers, helping to promote empirical research in this field. Researchers can use PTR to compare different models' performance and analyze their strengths and limitations in political reasoning tasks.

Model Development

For LLM developers, PTR offers a targeted test suite for:

• Identifying weaknesses in political temporal reasoning
• Guiding data selection and training strategies for model fine-tuning
• Validating the effectiveness of improvement measures

Policy Analysis

In policy research, models evaluated by PTR can serve as auxiliary tools to help analysts:

• Track the historical context of policy evolution
• Predict potential impacts of policy changes
• Identify association patterns between different political entities

PTR is developed in Python and uses Poetry for dependency management. Quick start steps:

1. Clone the repository: git clone https://github.com/iguillenp/ptr.git
2. Install dependencies: Use Poetry to install project dependencies
3. Run experiments: Execute the experiments.sh script to reproduce benchmark experiments
4. Explore data: Open Jupyter Notebooks for interactive analysis

Docker support is also provided for quick deployment across different environments.

PTR represents a useful attempt to combine knowledge graphs with LLM evaluation. By building a structured political temporal knowledge graph and designing targeted evaluation tasks, it provides a new approach for assessing LLMs' domain-specific reasoning capabilities.

In the future, the framework is expected to further expand to support more types of political reasoning tasks, learn from knowledge graph evaluation methods in other fields, and promote the overall development of LLM evaluation methodologies. For researchers and developers interested in political text analysis, temporal reasoning, and knowledge graph applications, PTR is an open-source project worth paying attention to and participating in.