Bibliothèque Française LLM: A French Public Domain Literature Index System Optimized for Large Language Models

Bibliothèque Française LLM is a structured indexing and annotation project for French public domain literature designed specifically for large language models (LLMs). It integrates multiple authoritative sources such as DraCor, Common Corpus, and Wikisource, providing metadata indexing categorized by genre, author, and era, as well as in-depth annotations for dramatic texts (including characters, lines, stage directions, etc.). Its aim is to enable LLMs to efficiently read and understand classic French literary works.

With the improvement of LLMs' text understanding, generation, and analysis capabilities, researchers are exploring AI applications in literary research, digital humanities, and cultural heritage preservation. However, existing digitized literature faces issues like inconsistent formats, missing metadata, and complex access interfaces, which hinder the effective use by LLMs. French literary heritage is rich but scattered across different platforms, with varying formats and a lack of unified indexing, limiting its usability. This project was thus born to optimize a structured indexing and annotation system for LLMs.

Core Philosophy: Mode Histoire: Create a system that allows LLMs to navigate, read, and interpret French literature in a 'historical reading' mode, emphasizing structured indexing, in-depth annotation, format optimization, and rich metadata.

Six Authoritative Sources:

1. Common Corpus (Pleias): 110 billion words of high-quality corpus;
2. French-PD-Books (Pleias): 289,000 books (16.4 billion words, requiring OCR correction);
3. DraCor—fre: 1,560 French plays with TEI annotations including characters, lines, etc.;
4. Wikisource: 50,000 manually proofread documents;
5. Project Gutenberg: Approximately 40,000 classic French works;
6. Ebooks libres et gratuits: 2,500 high-quality works (no API).

Modular architecture ensures full process traceability:

• Index Layer: Store metadata (genre, author, era, etc.) in Parquet/JSONL format;
• Source Layer: Extraction scripts for various data sources (DraCor API client, Common Corpus processor, etc.);
• Annotation Layer: Annotations for plays (characters, lines, stage directions, etc.)—novel annotations are in planning;
• Format Layer: LLM-optimized formats (Markdown, JSONL, TEI XML);
• Tool Layer: Tools for post-OCR processing, format conversion, text standardization, etc.

Application Scenarios:

1. Literary research assistance (quick text analysis, e.g., changes in the proportion of lines spoken by female characters);
2. Digital humanities teaching (lowering the threshold for research);
3. French language learning and LLM training;
4. Cultural heritage preservation (establishing digital archives).

Roadmap:

• Phase 1: Infrastructure setup (connecting DraCor API, defining index schema, etc.);
• Phase 2: Data integration and cleaning (importing Wikisource texts, OCR correction, etc.);
• Phase 3: LLM optimization and tool development (fine-tuning datasets, knowledge graphs, intelligent retrieval interfaces, etc.).

Challenge 1: Inconsistent OCR Quality → Multi-level quality control (automatic scoring, manual verification, community crowdsourcing); Challenge 2: Differences Between Classical and Modern French → Spelling normalization tools (mapping to modern forms while preserving original traceability); Challenge 3: Incomplete Metadata → Cross-validation and completion (combining multiple sources + external knowledge bases like Wikidata); Challenge 4: Complex Annotation for Genre Diversity → Scalable annotation schema (genre-specific annotation layers while maintaining consistent core metadata).

Open Source Strategy: Source texts are in the public domain; indexes and annotations use CC-BY-SA or equivalent open licenses to encourage collaboration.

Conclusion: This project represents a new direction for cultural heritage digitization in the AI era. It not only digitizes literature but also makes it usable for AI, promising to open up new possibilities for humanities research, language learning, and cultural preservation. We look forward to more achievements and applications.