LLM-HPC-Course: Practical Course on Distributed Training and Inference of Large Models on High-Performance Computing Platforms

LLM-HPC-Course is an open-source course project developed by HichamAgueny, designed for HPC environments, systematically explaining distributed training and inference of large models on supercomputing clusters. Using PyTorch as the framework and LLaMA model as the core case, the course covers distributed training, model fine-tuning, text summarization, and question-answering tasks, helping researchers and engineers efficiently conduct LLM research.

Course Background

The training/inference of large language models requires exponentially growing computing resources, which single-machine multi-card setups can hardly meet; HPC platforms have become important infrastructure due to their parallel computing capabilities and high-speed networks, but migration faces challenges such as parallel strategies and communication optimization.

Target Audience

• LLM researchers at supercomputing centers
• AI engineers expanding model training to multi-node setups
• Distributed deep learning learners
• HPC system administrators

The course is divided into 5 major modules:

1. HPC Environment Basics: Cluster architecture, SLURM scheduling, environment configuration, data management
2. Distributed Training Basics: PyTorch's DDP, model/pipeline/tensor parallelism
3. LLaMA Fine-Tuning Practice: Model quantization, LoRA fine-tuning, instruction fine-tuning, checkpoint management
4. Downstream Task Applications: Text summarization, question-answering systems, inference optimization
5. Performance Optimization and Debugging: Communication/memory/I/O optimization, performance analysis

Practice-Oriented

Each module is equipped with runnable code, sample datasets, SLURM script templates, and performance benchmark tests.

HPC Scenario Optimization

Integrates MPI to adapt to traditional supercomputers, optimizes multi-node communication (InfiniBand), solves storage I/O bottlenecks, and includes fault-tolerant design (automatic checkpointing).

Modular Design

Learners can skip modules as needed, and the code is independent for easy reuse and modification.

Advantages of Training LLMs on HPC

High cost-effectiveness, high-speed interconnection network, exclusive resource access, data security and compliance.

DeepSpeed ZeRO Optimization

ZeRO-1 (Optimizer state sharding), ZeRO-2 (Gradient sharding), ZeRO-3 (Parameter sharding), ZeRO-Offload (CPU/NVMe offloading).

Flash Attention

IO-aware block computation reduces complexity and decreases HBM access to improve throughput.

Path for Beginners (4-6 weeks)

Learn in module order: HPC Environment → Distributed Basics → LLaMA Fine-Tuning → Downstream Tasks → Performance Optimization.

Path for Experienced Learners (1-2 weeks)

Focus on HPC-specific content (Modules 1 and 5), directly run the fine-tuning process and modify configurations.

Community Feedback

• Fills the gap in HPC+LLM tutorials
• Clear code structure and easy to modify
• Practical SLURM script templates

Application Cases

• Graduate training courses at university supercomputing centers
• Domain-specific large model pre-training in research institutes
• Enterprises improving internal training frameworks

LLM-HPC-Course is a high-quality open-source project that systematically solves the problem of large model training on supercomputers and provides a complete path from theory to practice. It is recommended for those who need to carry out LLM work in HPC environments to practice hands-on with official documents and code to master relevant skills.