When Large Language Models Meet Arithmetic Coding: A New Breakthrough in Hybrid Text Compression on Distributed GPUs

The SMU research team has for the first time systematically evaluated hybrid compression schemes combining Transformer models such as BERT, RoBERTa, T5, and Llama with arithmetic coding, achieving scalable and efficient text compression on the NVIDIA DGX SuperPOD. This study fills the gap in benchmarking hybrid LLM+arithmetic coding schemes in distributed high-performance computing environments, and has open-sourced a complete reproducible codebase, providing valuable empirical data and tools for the field of neural network compression.

Text data is growing explosively, creating an urgent need for efficient storage and transmission. Traditional compression algorithms (e.g., gzip, bzip2) struggle to handle complex language structures, while large language models (LLMs) can capture long-range dependencies, understand context, and accurately predict token probability distributions. Arithmetic coding can theoretically reach the information entropy limit, but its performance depends on the accuracy of the probability model. Previously, there was a lack of systematic research on the performance and scalability of hybrid LLM+arithmetic coding schemes in distributed HPC environments.

Fine-tuning Phase

Four Transformer models (BERT, RoBERTa, T5-Small, Llama-3.2-3B) were fine-tuned using the enwiki9 dataset. After tokenization, the training data was converted into a 64-token context window, and autoregressive learning was used to predict the next token, supporting distributed data parallel acceleration from 1 to 16 GPUs.

Inference Compression Phase

After tokenization, new text is input into the fine-tuned LLM to generate token probability distributions, which are converted into integer cumulative distribution functions (CDF) for use by the arithmetic encoder, encoding into a compressed bitstream; during decompression, the original sequence is losslessly reconstructed using the same probability distribution and bitstream.

The experiments were conducted on the NVIDIA DGX A100 SuperPOD, which includes 20 nodes. Each node has 8 A100 80GB GPUs, 128 CPU cores, with a total of approximately 1.64 PFLOPS of computing power and 52.5TB of storage, and 200Gb/s InfiniBand interconnection between nodes.

Evaluation metrics include: compression ratio, bits per character (BPC), bits per token (BPT), cross-entropy, perplexity, KL divergence, reconstruction accuracy; as well as system-level metrics such as wall-clock time, memory usage, and scaling efficiency.

1. The first public work to scale analysis of hybrid LLM+arithmetic coding schemes on a top-tier HPC platform, filling the benchmark gap;
2. Provides a complete reproducible codebase (including fine-tuning/inference scripts for four models and SLURM configurations), lowering the threshold for subsequent research;
3. Offers new ideas for scenarios such as large-scale text archiving, genomic data compression, and log storage, and can be extended to other modalities like code and structured data.

The project code has been open-sourced, using conda for environment management (create the environment via environment.yml, install PyTorch 2.10, Transformers 4.57, etc.). The repository is organized by model directories, containing fine-tuning/inference code and SBATCH scripts.

Reproduction suggestions: Download the enwiki9 dataset, select the SBATCH script according to the number of GPUs to submit the task, and refer to the startup commands and path configurations in the documentation.

The SMU team's work combines academic research and engineering practice, integrating Transformer models and arithmetic coding technology, and systematically evaluating them on a supercomputing platform, contributing empirical data and open-source tools to the field of neural network compression. With the improvement of LLM efficiency and the growth of hardware computing power, neural network-based compression methods are expected to move from prototypes to practical deployment, bringing new possibilities for data-intensive applications.