76.9M Parameter Lightweight Story Generation LLM: A Complete Practice from Zero Training to Fine-Tuning

This article introduces a lightweight decoder-only language model with only 76.9 million parameters, designed specifically for creative story generation. The project demonstrates how to complete the full process from pre-training to fine-tuning on the free version of Google Colab, providing a reproducible reference solution for large model practices in resource-constrained scenarios.

Training large LLMs usually requires huge computing resources, which sets a high threshold. This project originated from the needs of a university course assignment. The goal is to build a small model that can understand story structures and generate creative text under the computing constraints (time and GPU resources) of the free Colab version, verifying the feasibility of lightweight architectures in resource-constrained environments.

It adopts a pure decoder architecture, referencing the paper 'Attention Is All You Need' and Andrej Karpathy's GPT implementation tutorial. With approximately 76.9 million parameters, it balances training feasibility (completable in a single Colab session), inference efficiency (suitable for edge deployment), and scalability (supports scale adjustment).

Two-stage training:

1. Pre-training: Use classic literary works from Project Gutenberg (e.g., Moby-Dick) to build basic language understanding;
2. Fine-tuning: Use the Reddit WritingPrompts dataset (prompt-story pairs) to develop specific story generation capabilities.

Sample performance:

• Given prompts like "A man in a sinking ship" or "A woman hugging a child", it can generate creative content with correct grammar and consistent person/tense;
• Limitations: Long text logic needs improvement, occasional repetition or topic drift. This performance is beyond expectations for a 76.9 million parameter model.

• Educational value: The code is concise and clear, and the complete process is suitable for learning LLM principles;
• Prototype verification: Low-cost verification of application ideas such as story generation;
• Resource-constrained deployment: Low latency and low memory usage, suitable for edge devices/high concurrency scenarios.

The author welcomes community contributions. Improvement directions include:

1. Scale expansion (increase layers/dimensions);
2. Data augmentation (diverse genres and styles);
3. Instruction fine-tuning (improve prompt understanding);
4. Quantization deployment (mobile support);
5. Multilingual expansion (e.g., Chinese).

This project proves that LLM capabilities come not only from parameter scale but also from reasonable architecture and training strategies. Although the 76.9 million parameter model cannot compete with 100-billion-level models, its performance on specific tasks is satisfactory, and it lowers the training threshold, promotes knowledge democratization, and is of great significance to learners and small teams.