Automatic Annotation of Ancient Constitutions Using Large Language Models: A New AI Tool for Political Science Research

The constitution_llm project is an open-source tool developed by deankuo. It uses a multi-model LLM pipeline to automatically analyze constitutional texts of historical regimes and extract 9 key political indicators, solving the problems of time-consuming manual annotation and poor consistency in traditional methods. It provides a reproducible AI tool for large-scale historical-political comparative research. The project is open-sourced on GitHub (link: https://github.com/deankuo/constitution_llm) and was released on 2026-06-01.

Comparative historical analysis in political science has long faced digital dilemmas: traditional manual annotation by experts is time-consuming and labor-intensive, making it difficult to ensure consistency and reproducibility across researchers, languages, and eras. Ancient constitutional texts, with their archaic languages, specific terminology, and uneven preservation status, further exacerbate the lack of standardized data infrastructure, becoming a bottleneck for research such as cross-civilization comparisons (e.g., institutional differences between the Roman Republic and the Han Empire).

The core features of the project include:

1. Multi-model support: Compatible with mainstream LLMs such as Google Gemini, OpenAI, Anthropic Claude, and AWS Bedrock; supports cross-model validation to improve result confidence;
2. Nine political indicator system: Defines dimensions such as sovereignty attribute (0/1 binary coding), federalism degree (binary), checks and balances mechanism (3-level scale), collegial system (binary), parliament type (4-level coding), appointment method (11 categories), removal method (16 categories), election competitiveness (3-level), etc., to achieve structured coding.

In terms of technical mechanisms:

• Validation strategies: Self-consistency validation (majority vote from 3 samples of the same input), validation chain (CoVe, cross-model cross-check of key indicators);
• Usage modes: Single prompt (quick test), multi-prompt (recommended production mode to reduce indicator interference), sequential prompt (test order impact);
• Efficiency optimization: Supports parallel processing to improve batch efficiency; the Gemini batch API can save 50% of costs.

The tool can be applied to various research scenarios:

• Institutional evolution research: Track long-term changes in indicators such as parliament type and election system;
• Cross-civilization comparison: Systematically compare political system characteristics of Europe, East Asia, the Islamic world, etc.;
• Democratization research: Build regime typology through appointment/removal methods;
• Federalism origin research: Identify early federalism experiments and test relevant theoretical hypotheses.

The project has the following limitations:

• Historical bias: LLM training data is mainly modern texts, which may lead to misinterpretation of pre-modern concepts;
• Language coverage: Processing capabilities for ancient languages (Latin, classical Chinese, etc.) need to be improved;
• Validation chain limitation: The current "--verify both" only runs CoVe, and sequential validation is not implemented. Future directions: Introduce historical language models, customize civilization-specific prompt templates, and build expert-annotated benchmark datasets.

constitution_llm is an important attempt at AI-assisted historical political research. It combines LLM capabilities with political science frameworks to provide a scalable solution for large-scale historical document processing. The tool is positioned as an auxiliary starting point for expert judgment, promoting digital humanities research through human-machine collaboration (automatic annotation + expert review). Its open-source nature allows the community to jointly improve it, contributing to methodological progress in the field.