Evaluation of Historical Italian LLMs: AI Challenges in Low-Resource Languages and Historical Texts

This project focuses on the evaluation of large language models (LLMs) for historical Italian, aiming to establish standardized evaluation benchmarks and reveal the unique challenges in AI for processing low-resource languages and historical texts. The research covers core content such as the complexity of historical languages, data scarcity issues, evaluation dimensions, technical methodologies, and application prospects.

Processing historical languages faces dual challenges: first, the inherent complexity of historical languages (spelling variations, grammatical evolution, vocabulary disappearance and evolution, differences in writing norms); second, the predicament of low-resource languages (limited digitized texts, scarce annotated data, few domain experts, insufficient research resources).

Italian was chosen as the research object due to its advantages such as a long written tradition, dialectal diversity, clear standardization process, and rich Renaissance literature. Historical Italian research covers three periods: Old Italian (1200-1300), Middle Italian (1300-1500), and the transition to modern times (1500-1800).

The evaluation may cover five dimensions: 1. Orthographic normalization (converting historical spelling to modern standards); 2. Morphological analysis (identifying ancient verb conjugations, noun declensions, etc.); 3. Syntactic analysis (processing historical word order and ancient sentence structures); 4. Semantic understanding (modern equivalents of ancient words, semantic evolution); 5. Chronological dating (judging the period of a text based on linguistic features).

The main technical challenge is data scarcity, with solutions including cross-language transfer learning, data augmentation, crowdsourced annotation, and digital humanities collaboration; evaluation standards need to be developed with the participation of domain experts, combining the characteristics of historical languages; model selection includes general multilingual models, Italian-specific models, historically fine-tuned models, and general LLMs.

The research significance is reflected in: digital humanities (large-scale literature analysis, author attribution, etc.); cultural heritage preservation (digitization of ancient books, translation and annotation, etc.); linguistic theory (verifying language evolution models); general NLP improvement (robustness enhancement, few-shot learning, etc.).

This project represents the direction of NLP research beyond high-resource languages: promoting technological democratization (reducing language inequality); facilitating interdisciplinary collaboration (integrating domain knowledge between NLP and humanities experts, co-building standard tools).

The relevant research ecosystem includes international projects (CLARIN, DARIAH, etc.), evaluation benchmarks (HistBERT, ICDAR competitions, etc.), and open-source tools (OCR4all, eScriptorium, etc.). The conclusion points out that this project is a bridge connecting the past and the future, promoting AI to develop in a more robust and inclusive direction.