LCATS: An Open-Source Tool System for Reconstructing Literary Text Analysis with Large Language Models

LCATS (Literary Captain's Advisory Tool System) is a comprehensive toolkit that combines traditional text processing techniques with modern large language model capabilities, supporting literary analysis, story extraction, and corpus research.

In the era of rapid AI development, large language models (LLMs) have demonstrated powerful text understanding and generation capabilities. However, when applying these capabilities to traditional humanities fields such as literary research and corpus analysis, researchers often face issues of fragmented tools and inconsistent workflows. LCATS (Literary Captain's Advisory Tool System) was created to address this pain point—it is a comprehensive toolkit that combines traditional text processing techniques with modern large language model capabilities.

LCATS was open-sourced by developer xenotaur, aiming to provide a one-stop solution for literary analysis, story extraction, and corpus-based research. The core concept of the system is to combine the intelligence of LLMs with the reliability of classic text processing methods to create a powerful yet interpretable literary research tool.

The project includes several carefully designed components:

• lcats Python package: Core library for text corpus creation and analysis
• Story Corpus: Public domain literary works collection organized in JSON format
• Analysis Tools: Text chunking, extraction, and story analysis functions
• Data Gatherers: Automatic data collection from sources like Project Gutenberg
• Processing Pipeline: Flexible multi-stage processing framework
• Command-Line Interface: Easy-to-use CLI supporting common operations

LCATS uses tiktoken for token-aware text segmentation, which is crucial for handling long novels or complex narrative texts. Traditional character-count-based segmentation often breaks semantic integrity, while LCATS's intelligent chunking ensures each segment maintains understandable context.

This is one of LCATS's most distinctive features. Users can define extraction requirements via templates, and the system uses the OpenAI API to automatically extract structured information from stories. For example, it can extract story events, character relationships, emotional trends, etc., and output them in JSON format for subsequent analysis.

The project has built-in a large number of public domain literary works, covering multiple classic authors:

• Andersen: Classic fairy tales and stories
• Brothers Grimm: German traditional folk tales
• Conan Doyle: Sherlock Holmes detective series
• Chesterton: Father Brown detective stories
• Lovecraft: Cthulhu Mythos series
• O. Henry: Short stories famous for unexpected endings
• Wilde: Literary works including The Happy Prince
• Jack London: Adventure and naturalist novels
• Hemingway: Modernist short stories
• Wodehouse: Humorous novels

Each work is stored in a unified JSON structure, containing complete metadata such as title, text, author, year, source URL, etc.

LCATS adopts a modular design, with core code located in the lcats/ directory:

• stories.py: Definitions of story and corpus classes
• pipeline.py: Processing pipeline framework
• chunking.py: Text chunking tools
• extraction.py: LLM-based data extraction
• analysis/: Text analysis and metric calculation
• gatherers/: Data collection modules
• cli.py: Command-line interface

The project is developed using Python 3.6+, with dependency management done via pyproject.toml. For users who need to use LLM functions, an OpenAI API key needs to be configured.

LCATS has a wide range of application scenarios:

Academic Research: Literary researchers can use LCATS to quickly build corpora on specific topics or authors for large-scale text analysis. For example, analyzing the frequency of specific imagery in literary works of a certain period, or tracking the evolution of narrative patterns.

Creative Writing: Writers and screenwriters can use the story extraction function to analyze the structure of classic works and learn narrative techniques. By comparing the stylistic features of different authors, they can gain creative inspiration.

Educational Applications: Teachers can use the built-in classic literary works library to design comparative reading assignments for students. The system supports multi-dimensional filtering by author, genre, era, etc., facilitating curriculum design.

AI Training Data Preparation: For AI projects that require high-quality literary texts as training data, LCATS provides ready-to-use corpora that have been cleaned and structured.