Open-Source Instruction Tuning Training Pipeline: A Complete Practical Solution from LoRA to DeepSpeed

The open-source project instruction-tuning-llm introduced in this article is a modular and configurable LLM training framework. It supports parameter-efficient fine-tuning methods like LoRA and QLoRA, integrates DeepSpeed distributed training and assistant-specific loss calculation, and provides developers with a flexible instruction tuning solution. The project focuses on instruction tuning and plans to expand to more post-training methods such as RLHF and DPO in the future.

Full-parameter fine-tuning of large language models consumes enormous resources, making Parameter-Efficient Fine-Tuning (PEFT) technology a necessity. This project is positioned as a configurable language model post-training pipeline, with core design philosophies of modularity and configurability: the training process is split into four modules—model loading, data processing, training engine, and distributed environment—all driven by YAML configuration files, allowing adaptation to different scenarios without modifying code.

The project supports mainstream PEFT methods: ① LoRA: Adds low-rank matrices alongside pre-trained model weights, training only the newly added parameters. Configuration can be specified via the peft_config in train.yaml, and it supports merging adapter weights to generate deployable models; ② QLoRA: Based on LoRA + 4/8-bit quantization, significantly reducing memory requirements (70B models can be fine-tuned on a single consumer-grade GPU). The project supports explicit control over adapter training precision (enable_lora_fp32 option).

Project's featured function—Assistant-Specific Loss: By masking non-assistant tokens, only the assistant's response tokens contribute to gradient updates, achieving more precise training objectives. This function relies on Jinja2 chat templates with specific tags to identify assistant responses. The data format uses JSONL (following OpenAI conversation standards, each data entry contains a messages array), supporting preprocessing such as automatic tokenization, length truncation, and dynamic batching.

The project integrates the DeepSpeed framework and supports ZeRO optimization stages 0-3: Stage0 is standard data parallelism (recommended for single GPU); Stage1 shards optimizer states; Stage2 shards gradients; Stage3 shards model parameters (recommended for multi-GPU large models). Distributed configuration is managed via Accelerate. Currently, it supports single-node multi-GPU training, and multi-node and FSDP support are planned.

The code structure is clear: main.py is the entry point, engine.py encapsulates SFTTrainer, model.py loads models/tokenizers, data.py processes data, and distributed.py and ds_utils.py handle distributed and DeepSpeed. Usage flow: ① Install dependencies (Flash Attention needs to be installed separately); ② Adjust configuration files in the configs directory; ③ Run the script (use run_single_gpu_train.sh for single GPU, run_multi_gpu_train.sh for multi-GPU).

Future plans: Add preference tuning methods (DPO, ORPO), reinforcement learning methods (PPO, GRPO), and support FSDP and multi-node training. Project value: Focused on instruction tuning, it provides reasonable default configurations, clear code structure, and detailed documentation. It is a practical starting point for developers to fine-tune LLMs and can flexibly adapt to professional assistant training or task customization needs.