Building Large Models from Scratch: A Complete Hands-On Tutorial with 23 Jupyter Notebooks

The GitHub project walkinglabs/modern-llm-notebook is a systematic LLM learning resource. It guides learners to implement core LLM components (Tokenizer, Attention, MoE, RLHF, inference acceleration, etc.) from scratch using PyTorch through 23 independent Jupyter Notebooks. It bridges the gap between calling APIs and understanding the internal mechanisms of models, following the teaching cycle of "intuitive understanding → manual calculation verification → code implementation → experimental observation".

Most LLM tutorials on the market stay at the application level (writing prompts, calling APIs, building RAG), lacking in-depth understanding of the model's essence. The core concept of this project is "handwriting core algorithms", enabling learners to not only know what works but also why it works.

The tutorial consists of 5 parts with 23 Notebooks:

1. Basic Architecture: Tokenizer, BPE, Embedding, Attention, Mini-GPT
2. Training Optimization: Architecture improvements (key LLaMA improvements), MoE, BERT, training loop, Scaling Laws, data engineering, LoRA, CPT, RLHF
3. Inference Acceleration: Generation strategies, KV Cache/FlashAttention, speculative decoding
4. Cutting-Edge Directions: Long context, CoT, VLM
5. Production Deployment: Evaluation, knowledge distillation, online policy distillation Each Notebook is self-contained, allowing learners to jump to the desired content as needed.

1. Manual Calculation Verification: Core algorithms are first calculated manually (e.g., MoE Router examples) to ensure understanding of mathematical meanings;
2. Alignment with Real Models and Papers: Covers models like GPT-4, LLaMA3, Mixtral, and more than 20 classic/latest papers;
3. Technical Details: Only relies on PyTorch (no encapsulated libraries like transformers). Environment requirements: Python3.9+, PyTorch2.0+, 16GB RAM; some chapters require a GPU. A web reader is provided.

Target Audience: Developers with PyTorch basics, AI researchers, algorithm engineers, technical managers; Learning Path: Quick Start (Part1) → Training Direction (Notebooks related to Part2) → Inference Optimization (Part3) → Cutting-Edge Exploration (Parts4-5).

This project fills the gap in LLM education. It is neither a collection of pure theoretical papers nor a superficial API tutorial, but a hands-on practice guide. In the era of rapid AI iteration, the ability to write core algorithms from scratch is a litmus test to distinguish ordinary users from experts, making it suitable for learners who want to deeply understand the working principles of LLMs.