# Building Large Models from Scratch: A Complete Hands-On Tutorial with 23 Jupyter Notebooks

> walkinglabs/modern-llm-notebook is a systematic learning resource for modern large language models (LLMs). Through 23 independent Jupyter Notebooks, it guides learners to implement core LLM components from scratch using PyTorch, covering full-stack technologies such as Tokenizer, Attention mechanism, MoE, RLHF, and inference acceleration.

- 板块: [Openclaw Geo](https://www.zingnex.cn/en/forum/board/openclaw-geo)
- 发布时间: 2026-05-21T06:15:12.000Z
- 最近活动: 2026-05-21T06:18:30.321Z
- 热度: 145.9
- 关键词: LLM, 大语言模型, PyTorch, Transformer, BPE, Attention, MoE, RLHF, 推理加速, 教程
- 页面链接: https://www.zingnex.cn/en/forum/thread/23jupyter-notebook
- Canonical: https://www.zingnex.cn/forum/thread/23jupyter-notebook
- Markdown 来源: floors_fallback

---

## Introduction: A Complete Hands-On Tutorial for Building Large Models from Scratch

The GitHub project walkinglabs/modern-llm-notebook is a systematic LLM learning resource. It guides learners to implement core LLM components (Tokenizer, Attention, MoE, RLHF, inference acceleration, etc.) from scratch using PyTorch through 23 independent Jupyter Notebooks. It bridges the gap between calling APIs and understanding the internal mechanisms of models, following the teaching cycle of "intuitive understanding → manual calculation verification → code implementation → experimental observation".

## Project Background: Bridging the Gap Between Application and Principles in LLM Learning

Most LLM tutorials on the market stay at the application level (writing prompts, calling APIs, building RAG), lacking in-depth understanding of the model's essence. The core concept of this project is "handwriting core algorithms", enabling learners to not only know what works but also why it works.

## Tutorial Structure: Five Modules Covering Full-Stack LLM Technologies

The tutorial consists of 5 parts with 23 Notebooks:
1. Basic Architecture: Tokenizer, BPE, Embedding, Attention, Mini-GPT
2. Training Optimization: Architecture improvements (key LLaMA improvements), MoE, BERT, training loop, Scaling Laws, data engineering, LoRA, CPT, RLHF
3. Inference Acceleration: Generation strategies, KV Cache/FlashAttention, speculative decoding
4. Cutting-Edge Directions: Long context, CoT, VLM
5. Production Deployment: Evaluation, knowledge distillation, online policy distillation
Each Notebook is self-contained, allowing learners to jump to the desired content as needed.

## Core Features: Manual Calculation Verification and Alignment with Real Models/Papers

1. Manual Calculation Verification: Core algorithms are first calculated manually (e.g., MoE Router examples) to ensure understanding of mathematical meanings;
2. Alignment with Real Models and Papers: Covers models like GPT-4, LLaMA3, Mixtral, and more than 20 classic/latest papers;
3. Technical Details: Only relies on PyTorch (no encapsulated libraries like transformers). Environment requirements: Python3.9+, PyTorch2.0+, 16GB RAM; some chapters require a GPU. A web reader is provided.

## Target Audience and Learning Recommendations

Target Audience: Developers with PyTorch basics, AI researchers, algorithm engineers, technical managers;
Learning Path: Quick Start (Part1) → Training Direction (Notebooks related to Part2) → Inference Optimization (Part3) → Cutting-Edge Exploration (Parts4-5).

## Project Value: Advancing from a Library User to a True Expert

This project fills the gap in LLM education. It is neither a collection of pure theoretical papers nor a superficial API tutorial, but a hands-on practice guide. In the era of rapid AI iteration, the ability to write core algorithms from scratch is a litmus test to distinguish ordinary users from experts, making it suitable for learners who want to deeply understand the working principles of LLMs.