EduAssistGPT: A Domain-Specific Large Language Model for Education Based on TinyLlama and LoRA

EduAssistGPT is an education domain-specific large language model chatbot project based on TinyLlama-1.1B-Chat and LoRA parameter-efficient fine-tuning technology. It demonstrates the complete development lifecycle of a domain-specific LLM from data preparation to deployment, aiming to address the pain points of traditional education support systems and enhance the domain depth and accuracy of educational AI.

Traditional education support systems face limitations such as limited teacher time, delayed student responses, and lack of personalized guidance, which are particularly prominent in online education scenarios. While general-purpose LLMs have broad knowledge, they lack domain depth and accuracy. Fine-tuning general models into domain-specific ones has become a key path to improve the quality of educational AI, leading to the birth of the EduAssistGPT project.

Base Model: TinyLlama-1.1B-Chat

Selection reasons: High resource efficiency (trainable on consumer-grade GPUs), fast training speed, deployment-friendly, and good foundation for dialogue optimization.

Fine-tuning Technology: LoRA

Advantages: Freezes original model weights, only trains low-rank matrices, has few trainable parameters (<1%), low GPU memory requirements, short training time, small storage overhead, and easy multi-task support.

Dataset: SQuAD

Features: Manually generated questions and verified answers, rich contextual information, with 87,599 samples in the training set and 10,570 samples in the validation set.

1. Data Preprocessing and EDA: Analyze dataset structure, missing values, length distribution, etc., and confirm the high quality of SQuAD.
2. Instruction Dataset Construction: Convert question-answer pairs into instruction format (instruction + context + question + answer).
3. Training Configuration: Epochs=1, Batch Size=2, Learning Rate=2e-4, Training Samples=5000, GPU=Tesla T4, final training loss=0.395.
4. Gradio Deployment: Build a simple interactive web interface that supports real-time responses and public sharing.

Function Demonstration

Can answer educational questions such as: "What is machine learning?" "What is artificial intelligence?" and other core AI concept questions.

Application Scenarios

• Instant learner support (24/7 response)
• Improve educational accessibility (serve remote areas)
• Reduce teacher burden (handle basic questions)
• Support self-directed learning (answer questions anytime)
• Enhance learning engagement (instant feedback)

• Large-scale dataset processing: Efficiently load 87K+ samples using Hugging Face Datasets
• Instruction tuning data preparation: Design reasonable instruction templates
• GPU memory limitations: Train on T4 GPU using LoRA and gradient checkpointing
• LoRA configuration tuning: Experiment with different ranks and scaling parameters
• Model deployment: Simplify the process with Gradio for quick effect verification

Future Improvement Directions

• Full dataset training
• RAG integration (external knowledge base reference)
• Multilingual support
• Textbook integration
• Cloud platform deployment
• Advanced tutoring functions (personalized path recommendation)

Project Significance

• Demonstrate the value of parameter-efficient fine-tuning and lower the threshold for AI development
• Emphasize the importance of high-quality domain data
• Prove the potential of lightweight models in specific domains
• Provide an end-to-end reusable methodology and serve as a reference for domain-specific AI assistant development