HRM-Text: An Open-Source Framework for Pre-training 1B-Parameter Large Language Models on a $1000 Budget

The HRM-Text project demonstrates how to pre-train a 1-billion-parameter foundation model from scratch for approximately $1000. By using a hierarchical recurrent architecture (HRM) and efficient data engineering, it reduces computational requirements by 130-600 times, providing a feasible path for democratizing large model pre-training. The project open-sources a full-process toolchain and supports multi-size model configurations.

Traditional large models (e.g., GPT-4, Claude) require tens of millions of dollars for pre-training, needing thousands of top-tier GPUs to run for months—only tech giants can participate. While there are technologies like Parameter-Efficient Fine-Tuning (PEFT), they rely on existing pre-trained models, and pre-training from scratch remains resource-intensive. HRM-Text breaks this situation, proving that a $1000 budget can pre-train a competitive 1-billion-parameter model.

Architectural Innovation: Adopts HRM hierarchical recurrent design (H module handles long-range dependencies, L module processes local features, recursive reasoning transfers information); PrefixLM hybrid training paradigm (prefix bidirectional attention, causal attention for the generation part); integrates FlashAttention3 optimizations (IO-aware scheduling, block-wise computation, fused kernels).

Data Engineering: Multi-stage cleaning (quality filtering, deduplication, tokenization); hierarchical sampling strategy (domain balance, difficulty scheduling, deterministic sampling) to improve data utilization efficiency.

HRM-Text provides two recommended configurations:

• L Configuration (0.6B parameters): 8 H100 GPUs on a single node, 50 hours, cost ~$800; GSM8k 77.6%, MATH51.2%, MMLU56.6%, HellaSwag52.7%.
• XL Configuration (1B parameters): 16 H100 GPUs on two nodes, 46 hours, cost ~$1472; GSM8k 84.7%, MATH56.5%, MMLU60.7%, HellaSwag63.4%.

These results are achieved with limited resources, reflecting the efficiency advantages of the architecture.

The project provides a full-process open-source toolchain:

• Training Infrastructure: PyTorch FSDP2 distributed training, Docker images (including CUDA/PyTorch dependencies), Weights & Biases integration.
• Evaluation & Export: Multi-benchmark testing (GSM8k/MATH, etc.), Hugging Face format export, vLLM inference acceleration (under development).
• Baseline Comparison: Implementations of standard Transformer, TRM, RINS, Universal Transformer, etc., to facilitate comparative experiments.

Significance of HRM-Text:

1. Lowering Threshold: Reduces pre-training costs to $1000, allowing academic institutions and individual researchers to participate in foundation model innovation and break monopolies.
2. Validating Architectural Value: Proves that architectural innovation (not just parameter stacking) can improve efficiency, challenging the research direction dominated by Scaling Law.
3. Data Engineering Demonstration: Hierarchical sampling and domain balance strategies provide practical experience for the community.

Limitations: The current maximum validated scale is 1B parameters; the effectiveness of larger scales remains to be verified; it is mainly English-oriented, and multilingual capabilities need to be expanded; the advantages of long-context modeling require more experiments.

Future Directions: Expand to larger parameter scales; develop multilingual versions; integrate efficient fine-tuning technologies; customize training for specific domains (code, scientific literature).