Large-Scale Pre-Training Practice of Deep Recurrent Language Models: Technical Analysis of the Huginn Project

The Huginn project successfully completed large-scale pre-training of deep recurrent language models on 4096 AMD GPUs, exploring the feasibility of alternative architectures to Transformers. This article covers model architecture design, distributed training strategies, AMD platform optimization techniques, and inference deployment solutions, providing valuable engineering practice references for the direction of deep recurrent models.

Traditional Transformer models improve capabilities by increasing the number of layers, but memory and computing costs grow exponentially. Deep recurrent models achieve deeper computational paths with a fixed parameter scale by allowing information to flow repeatedly within the same layer, and can expand computational volume by increasing the number of cycles during testing. The Huginn project, based on the report "Scaling up Test-Time Compute with Latent Reasoning", verifies the large-scale feasibility of deep recurrent architectures, and choosing AMD GPU clusters brings unique technical challenges and value.

The core innovation of deep recurrent models is converting physical layers into the number of computational cycles, applying the same set of parameters multiple times. Its features include: parameter efficiency (achieving complex paths with fewer parameters), scalable test-time computation (increasing the number of cycles after training without retraining), and implicit reasoning ability (cycles equal multi-step reasoning). The huginn-0125 model adopts the nebel-raven-3.5b architecture, estimated to be at the 3.5 billion parameter level, and its effective computational capacity may exceed that of Transformers with the same number of parameters.

Training on 4096 GPUs faces gaps in the AMD software ecosystem (ROCm vs CUDA), especially in large-scale cluster communication. The code was heavily modified based on LitGPT, with core logic in train.py and model definition in recpre/model_dynamic.py. The _allreduce_chunk_stream method of the key component SimpleFabric class (recpre/utils.py) solves the RCCL communication hang problem, and streaming block communication bypasses the all-reduce deadlock of the OFI plugin.

Environment configuration is managed via launch_frontier.py, including fine-tuning of numerous AMD system environment variables. Data parallelism strategy: 4096 parquet files correspond to the number of GPUs, each sample has 4096+1 tokens, local micro-batch size is 1, and one line is read from each file per step. The author does not directly recommend the code but hopes to provide a reference for large model training on AMD systems, emphasizing that successful operation takes priority over code elegance.

The training data preparation process is complete; the tokenizer is generated using scripts/tokenizer_generation.py (dependent on the bpeasy BPE trainer), and data download is done via scripts/scalable_data_download.py (the author acknowledges that it is time-consuming, space-intensive, and error-prone). The training dataset is uploaded to Hugging Face, containing 4096 parquet files for training and validation, with open data available for community reproduction and improvement.

Inference solutions include a concise reference implementation recpre/raven_modeling_minimal.py (compatible with Hugging Face) and vLLM-accelerated inference (supporting batch processing and memory management). Evaluation is compatible with lm-eval harness; code tasks are executed using bigcode, and the evaluation command includes mean_recurrence=32 (default 32 cycles). The GSM8k task requires specific system prompts and chat formats to achieve optimal results.

Huginn demonstrates the large-scale feasibility of deep recurrent architectures and provides practical references for large model training on non-NVIDIA platforms. The computational expansion capability of recurrent models during testing (trading off resources and output quality) has application value. The project provides detailed reproduction steps to promote progress in the field. Whether deep recurrent models will become competitors to Transformers remains to be tested, but this path is worth exploring and may provide a way to expand model capabilities beyond parameters.