LLM Creation Kit: Train Your Own Large Language Model on Consumer GPUs

LLM Creation Kit is a complete Python toolkit that allows developers to train their own large language models from scratch on consumer hardware (e.g., RTX 4070), supporting multiple configurations from 30M to 1.5B parameters. The project adopts modern architectural design (RoPE positional encoding, RMSNorm normalization, GQA attention, MoE structure), aligns with mainstream model technologies, and also provides features like an interactive training wizard, inference model support, and model export/deployment.

LLM training was once considered a patent of tech giants, requiring massive computing clusters and funds. LLM Creation Kit changes this situation by supporting training on consumer hardware (e.g., RTX 4070 with 12GB VRAM), covering parameters from 30 million (smoke test) to 1.5 billion (flagship level), and its architecture is aligned with mainstream models like LLaMA-2/3 and Mixtral.

• Core Components: Uses RoPE positional encoding (better length generalization), RMSNorm Pre-Norm structure (stable and efficient training), GQA attention (reduces inference KV cache);
• MoE Architecture: The 1.5B parameter model only activates about 25% of FFN parameters, achieving large model capacity at the cost of a small model;
• Other Technologies: SwiGLU activation function, GPT-2 BPE tokenizer, weight tying (reduces parameters by 10%), 8-bit AdamW optimizer (reduces VRAM usage by 75%).

The project provides an interactive TUI wizard via kit.py with an 8-step configuration process:

1. Model type selection (standard/inference model);
2. Model size selection (preset or custom);
3. Dataset selection (built-in or custom);
4. Hyperparameter adjustment (smart defaults + fine-tuning);
5. Early stopping settings;
6. Advanced options (8-bit AdamW, torch.compile, etc.);
7. Context length setting;
8. Output configuration. Supports exporting configurations to YAML for reuse, and training can be resumed via --load after interruption.

Six preset sizes optimized for hardware constraints:

Preset	Parameter Count	VRAM Requirement	Training Time on RTX4070	Context Length
30m	30M	~2GB	~10 minutes	512
70m	70M	~3GB	~1 hour	1024
125m	125M	~5GB	~8 hours	1024
350m	350M	~8GB	~2 days	2048
1b	1B	~10GB	~1 week	2048
1.5b	1.5B	~12GB	~3 weeks	2048
For models with 1B+ parameters, it is recommended to enable --use_8bit_adam (reduces optimizer VRAM usage by 75%), and gradient checkpointing is automatically enabled.

• Inference Models: Training data needs to include <thinking> (reasoning process) and <answer> (final answer) tags. Built-in inference datasets like GSM8K and MetaMathQA are provided, and a two-stage strategy of pre-training + fine-tuning is recommended;
• Generation Features: generate.py supports single-prompt generation, multi-completion sampling (--n parameter), and interactive dialogue (--interactive). For inference models, whether to show the chain-of-thought can be controlled via --show_thinking.

• Export and Deployment: Convert to GGUF format via convert_gguf.py (supports quantization like f16/q8_0/q4_k_m), and can be integrated with Ollama;
• Training Monitoring: Supports Weights & Biases to record metrics like loss and learning rate;
• Recovery Mechanism: --resume to restore training from checkpoints, with built-in early stopping mechanism to prevent overfitting.

LLM Creation Kit is an open-source project that enables developers with consumer GPUs to train LLMs through preset configurations, interactive wizards, and modern architecture. Its value lies in conveying the idea that LLM training is not just a patent of giants—individual developers and small teams can also participate in innovation, providing a solid starting point for this vision.