Implementing Qwen 3.5 Inference from Scratch: A Deep Learning Project on CUDA and LLM Principles

This is an open-source educational project maintained by TKristof09 on GitHub. It aims to help developers gain an in-depth understanding of CUDA programming and the underlying principles of LLMs by implementing the inference engine for Qwen3.5's dense architecture from scratch. The project does not rely on high-level frameworks like PyTorch; instead, it uses CUDA C++ to write core operators directly. Key content includes CUDA operator implementation, Qwen3.5 architecture analysis, memory optimization, etc. Project link: https://github.com/TKristof09/qwen-inference, last updated on 2026-07-12T17:22:07Z.

Most current LLM inference engines are encapsulated in high-level frameworks, leading to developers' lack of understanding of underlying mechanisms. This project fills the gap by 'building the wheel from scratch' (no frameworks, direct CUDA implementation). Reasons for choosing Qwen3.5: it is an open-source model by Alibaba with excellent Chinese language capabilities, combining practical application value and advanced architectural design.

CUDA Core Operator Implementation

• Matrix Multiplication (GEMM): Optimization from naive implementation to performance close to cuBLAS (memory access, shared memory, warp parallelism);
• Attention Mechanism: Core logic of Multi-Head Attention (MHA) and Grouped Query Attention (GQA);
• Layer Normalization: Variants like RMSNorm and numerical stability optimization;
• Activation Functions: Efficient CUDA kernels for SwiGLU and SiLU.

Qwen3.5 Dense Architecture Analysis

• Transformer Decoder layer stacking (pre-normalization design);
• Rotational Position Encoding (RoPE) CUDA implementation;
• KV cache management (inference acceleration);
• INT8/INT4 weight quantization support (reducing memory usage).

Memory Management Optimization

• Weight memory layout optimization (improving cache hit rate);
• Activation value reuse (reducing memory allocation overhead);
• Paged Attention (flexible KV cache management).

• System Performance Engineers: Understand inference performance bottlenecks and write efficient CUDA kernels;
• AI Infrastructure Developers: Gain insight into underlying mechanisms and optimize inference services (e.g., Triton, TensorRT-LLM);
• Deep Learning Researchers: Identify areas for framework improvement and propose optimization solutions;
• Computer Science Students: Combine parallel computing theory with industrial applications to enhance practical skills.

• CUDA Programming Complexity: Modular code + detailed comments to decompose complex kernels;
• Numerical Precision Control: Reasonable algorithm design + stability techniques to ensure output quality;
• PyTorch Weight Compatibility: Implement tensor format conversion and type mapping to load pre-trained weights;
• Performance Tuning: Profile-guided optimization to target and improve performance of hot kernels.

This project provides learning resources for LLM underlying principles in an open-source manner. Although 'bare-metal' implementation has low development efficiency, it has unique educational value. Developers share their learning process (code + documentation), contributing to community knowledge sharing and promoting the prosperity of the open-source ecosystem.

The qwen-inference project provides an excellent platform for in-depth understanding of LLM inference mechanisms, helping to establish a systematic understanding of key technologies such as Transformer architecture, CUDA programming, and memory optimization. It is recommended for technical personnel in AI infrastructure and inference optimization directions to try it out. The experience of 'building the wheel' can deepen understanding of tools and cultivate the ability to solve complex system problems.