# Building Large Language Models from Scratch: A Comprehensive Analysis of the LLMs-from-Scratch Project > This article provides an in-depth introduction to the LLMs-from-Scratch open-source project, which offers complete tutorials and codebases for implementing large language models (LLMs), vision-language models (VLMs), and multimodal models from the ground up. It covers the implementation of core technologies—including the Transformer architecture, attention mechanisms, and training pipelines—from scratch. - 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm) - 发布时间: 2026-03-30T17:45:04.000Z - 最近活动: 2026-03-30T17:50:06.910Z - 热度: 159.9 - 关键词: LLM, Transformer, PyTorch, 深度学习, 视觉语言模型, BPE分词, 注意力机制, 开源项目 - 页面链接: https://www.zingnex.cn/en/forum/thread/llms-from-scratch - Canonical: https://www.zingnex.cn/forum/thread/llms-from-scratch - Markdown 来源: floors_fallback --- ## 【Introduction】LLMs-from-Scratch Project: A Complete Guide to Building Large Language Models from Scratch # 【Introduction】LLMs-from-Scratch Project: A Complete Guide to Building Large Language Models from Scratch LLMs-from-Scratch is an open-source project created by developer Jkanishkha0305, designed to help learners understand and implement large language models (LLMs), small language models (SLMs), and vision-language models (VLMs) from scratch. The project covers the low-level implementation of core technologies such as the Transformer architecture, attention mechanisms, and training pipelines. By writing code hands-on, learners can deeply grasp the principles of model design rather than just staying at the usage level. ## Project Background and Core Objectives ## Project Background and Core Objectives The core philosophy of the LLMs-from-Scratch project is to help developers and researchers break the "black box" mystery of LLMs through the approach of "building from scratch". The project requires learners to start from basic components and gradually master the details of modern Transformer architectures, covering model implementations in three major domains: text, vision, and multimodality. The value of this learning method lies in not only learning how to use models but also understanding the logic behind their design, which is crucial for model optimization, troubleshooting, and innovative research. ## Detailed Explanation of Core Technology Implementations ## Detailed Explanation of Core Technology Implementations ### Transformer Decoder Architecture The project implements a causal Transformer architecture inspired by the LLaMA series, focusing on autoregressive text generation. Key technologies include: - **Multi-head attention**: Implements query, key, value projection calculations, scaled dot-product attention, and result concatenation; - **Rotary Position Encoding (RoPE)**: Injects relative position information to enhance generalization ability for long sequences. ### Normalization and Activation Functions Uses RMSNorm pre-normalization (a lightweight alternative to LayerNorm) and the SwiGLU activation function to effectively improve model performance. ### Optimization Strategies Implements weight sharing between input/output embedding layers (reducing parameter count) and uses KV caching during inference to reduce redundant computation overhead. ## Tokenization and Data Processing Pipeline ## Tokenization and Data Processing Pipeline ### Custom BPE Tokenizer Implements a Byte Pair Encoding (BPE) tokenizer from scratch, building subword units by iteratively merging high-frequency character pairs to balance vocabulary size and expressive power. ### Training Data Pipeline Includes cleaning, tokenization, and encoding processes for large-scale text corpora, and builds a custom iterable dataset loader that supports batching and efficient pipelining. ## Training Evaluation Strategies and Mixture of Experts (MoE) Models ## Training Evaluation Strategies and Mixture of Experts (MoE) Models ### Training and Loss Function Uses cross-entropy loss for next-token prediction training, which directly corresponds to the core task of language models. ### Evaluation and Sampling Supports perplexity calculation, loss trend tracking, and qualitative text analysis; implements Top-k and Top-p sampling techniques to balance creativity and controllability of generated text. ### Mixture of Experts (MoE) Architecture Explores MoE implementation: introduces expert network layers in feed-forward blocks, uses Top-K gating mechanism and load balancing loss to ensure even usage of experts, and implements a shared expert mechanism to provide baseline generalization ability. ## Vision-Language Models: PaliGemma and SigLip ## Vision-Language Models: PaliGemma and SigLip ### PaliGemma Implementation Uses a ViT encoder + Gemma decoder architecture for image caption generation; visual features are projected via a linear layer and decoded together with text tokens, supporting RoPE position encoding, RMSNorm, and Top-P sampling for visual question answering. ### SigLip Architecture A contrastive learning-based model for image-text pair processing, using a vision Transformer backbone network, paired with an independent text encoder and MLP, trained via cosine similarity loss and learnable temperature parameters. ## Technology Stack and Learning Value of the Project ## Technology Stack and Learning Value of the Project ### Technology Stack The project is built on Python and PyTorch, with key dependencies including: - PyTorch (core framework); - Hugging Face Datasets (pre-tokenized datasets); - Weights & Biases (experiment tracking); - Jupyter Notebooks (prototype development); - Matplotlib/Seaborn (visualization). ### Learning Value Through the project, you can gain: 1. A solid foundation in Transformer components; 2. Engineering practice skills for scalable training pipelines; 3. Model debugging and troubleshooting capabilities; 4. Principle-based innovative thinking. ## Conclusion and Open Source Notes ## Conclusion and Open Source Notes In today's era of rapid AI technology iteration, the LLMs-from-Scratch project provides an opportunity to deeply understand the underlying logic of LLMs. Whether you are a beginner or a professional, you can gain a deep intuition for the technology by building models with your own hands. The project is open-sourced under the MIT license, and community contributions and feedback are welcome.