Common Sense-Driven Fine-Tuning of Large Models: Enabling AI to Write More Coherent Stories

This article presents a common sense reasoning-based fine-tuning method for large language models. It adapts three Transformer models using LoRA technology and trains them on the ROCStories common sense dataset, significantly improving the logical coherence of story continuation tasks. The project is from GitHub, authored by nithin-jella, with the original title Commonsense-Driven-Fine-Tuning-of-Transformer-Models-for-Coherent-Story-Generation, link: https://github.com/nithin-jella/Commonsense-Driven-Fine-Tuning-of-Transformer-Models-for-Coherent-Story-Generation, published on 2026-06-12T22:41:36Z.

Large language models excel at text generation, but maintaining plot coherence and logical consistency in long text story generation remains a major challenge. Traditional models often produce content with contradictions and lack of causal relationships, rooted in their insufficient deep understanding of 'common sense'. Common sense reasoning is the foundation of human cognition, such as implicit knowledge like 'fire is hot' or 'the ground gets wet on rainy days', which AI struggles to capture effectively.

This project proposes a 'common sense-driven' fine-tuning method that closely integrates common sense reasoning with generation tasks to improve the model's logical consistency. Key technical features:

1. Multi-model architecture: Fine-tunes three Transformer models simultaneously and uses ensemble learning to enhance generation quality;
2. Efficient LoRA fine-tuning: Adopts low-rank adaptation technology to reduce computational resource requirements;
3. ROCStories dataset: Uses a short story dataset with common sense annotations for training.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method. It adds low-rank matrices next to the original pre-trained weights to adapt to tasks instead of modifying the original weights. Advantages:

1. High parameter efficiency: Only trains a small number of newly added parameters;
2. Modularity: Different LoRA adapters can switch tasks;
3. Preserves capabilities: Retains the original model's general capabilities while gaining task-specific expertise. This project uses LoRA to allow consumer-grade hardware to fine-tune large models, lowering the threshold.

ROCStories is a dedicated dataset for common sense reasoning, containing over 50,000 five-sentence daily stories, all manually verified to comply with common sense logic. Through training on this dataset, the model learns:

• To identify causal relationships in stories;
• To predict event developments that align with common sense;
• To avoid generating content that contradicts the physical world or social norms.

The project uses multi-dimensional standard evaluation metrics:

• BLEU score: Measures the n-gram overlap between generated text and reference text;
• ROUGE score: Evaluates text similarity based on recall;
• BERTScore: Uses BERT embeddings to calculate semantic similarity;
• Perplexity: Measures the model's confidence in generated text. These metrics comprehensively evaluate story quality, covering both surface text similarity and semantic coherence.

Application Scenarios:

• Creative writing assistance: Provides logical continuation suggestions for writers;
• Educational tools: Offers common sense feedback for language learning and writing practice;
• Content generation: Story prototypes for games and films;
• Dialogue systems: Improves contextual consistency in long conversations. Future Directions:
• Expand common sense knowledge bases (e.g., ConceptNet, ATOMIC);
• Explore multilingual common sense reasoning;
• Apply to tasks like code generation and mathematical reasoning.

Common sense-driven fine-tuning of large models is an important step for NLP towards 'smarter and more reliable' generation models. By combining the efficiency of LoRA and the common sense annotations of ROCStories, this project provides a reusable technical framework for story generation, which is valuable for researchers and developers aiming to improve the logical consistency of generation models.