From Zero to One: LLM Mechanisms Lab - A Deep Learner's Technical Notes Repository

This article introduces the GitHub repository llm-mechanisms-lab, a systematic LLM learning notes repository maintained by woithook. It combines theory and practice via Jupyter Notebooks, covering core technologies like BPE tokenization, Transformer implementation, RoPE positional encoding, and VLA multimodal models. Positioned as a personal learning memo, it showcases a real, progressive learning path to help learners uncover the "black box" of LLMs.

Large Language Models (LLMs) like GPT and Llama have transformed the AI field, but they remain a "black box" for many learners. woithook created the llm-mechanisms-lab repository to address this issue—it is not a project chasing SOTA performance, but a record of the learning process from theory to implementation via Jupyter Notebooks. The content is personal research memos (some translated/written with LLM assistance), showing a real learning path rather than a perfect tutorial.

The repository has bilingual directories en/ and zh/. The Chinese directory includes the following core topics:

1. BPE Tokenization Principles: Understanding tokenizers from the perspective of information compression, breaking down the training process, differences from Huffman coding, token boundary decisions when prefixes repeat
2. Implementing Large Language Models from Scratch: Character-level tokenization, baseline model design, self-attention mechanism implementation, Transformer Block construction, GPT-style model training and fine-tuning
3. RoPE and Transformer Memory: Mathematical principles of rotary positional encoding, long-sequence memory mechanisms and experimental verification
4. Revival of SSM (S4): Advantages in sequence modeling compared to Transformer
5. Introduction to VLA: Multimodal model concepts and architecture design
6. Robot Diffusion Policy: Application of diffusion models in robot control
7. Pre-training on WikiText: Complete workflow and practical details All content is presented in Jupyter Notebook format.

The BPE tokenization Notebook starts with an essential question: "Is BPE about understanding language or compressing strings?"

• Information Compression and Language Representation: BPE aims to find high-frequency adjacent segments in training corpus and merge them into new tokens, revealing that LLMs are based on statistical pattern matching rather than "understanding" language
• Difference from Huffman Coding: Huffman assigns long/short bit strings to existing symbols to optimize average length; BPE dynamically builds a vocabulary to change the tokenization method
• Prefix Repetition Decision: Token boundaries come from the priority of merging rules. For example, whether "learning" is split depends on whether there is a rule to merge "learn+ing", not on linguistic root/affix judgment.

This Notebook breaks down into five core questions:

1. How do computers understand text?: Build a character-level dictionary with bidirectional mapping between strings and integers
2. How to predict the next word?: A baseline model demonstrates the basic principles of language models (predicting the next token based on existing information)
3. How to go beyond short-sighted prediction?: Introduce self-attention mechanism to allow the model to review all relevant previous information
4. How to build a thinking brain?: Construct Transformer Blocks (self-attention + feed-forward network + residual connection + layer normalization) and stack them into a complete GPT model
5. How to adapt to different tasks?: Discuss methods for transferring pre-trained models to downstream tasks via fine-tuning

For Beginners:

1. Progressive learning path (from character-level tokenization to Transformer architecture)
2. Combination of theory and practice (each concept is accompanied by code implementation)
3. Fills the gap of high-quality Chinese LLM resources For Advanced Researchers:
4. In-depth exploration of details (e.g., RoPE memory analysis, BPE prefix decision)
5. Coverage of multiple topics (from traditional NLP to cutting-edge multimodal and robot learning)
6. Real thinking process (including "pitfall" records)

Limitations pointed out by the repository author:

1. Informal Publication: The content is personal notes, which may contain incomplete explanations, experimental code, or inaccurate translations
2. LLM-assisted Generation: Some content is translated/rewritten with LLM assistance, so terms, explanations, or annotations may not be standardized
3. Need for Careful Verification: Important technical details should be checked before use in research/teaching/engineering These reminders reflect the author's sense of responsibility and clarify the source and reliability of the content.

llm-mechanisms-lab demonstrates a learning approach of active exploration, recording, and experimentation. Each Notebook is problem-driven to delve into technical details. Core insights:

1. Understanding is more important than implementation (the difference between calling a library and truly understanding the architecture)
2. Start from first principles (BPE is information compression, self-attention is weighted context aggregation)
3. Value of recording and sharing (open-sourcing personal notes creates knowledge compounding) In today's era of rapid LLM development, such a "mechanisms lab" helps us see the essence of technology beyond the hype.