When Large Language Models Meet Arithmetic Coding: A New Paradigm for Text Compression on Distributed GPUs

The SMU research team open-sourced the first hybrid text compression system combining Transformer-based LLMs with arithmetic coding. It achieves multi-GPU distributed compression on the DGX A100 SuperPOD and supports four model architectures: BERT, RoBERTa, T5, and Llama-3.2-3B, bringing a new paradigm to the field of text compression.

Text compression essentially eliminates information redundancy. Traditional algorithms like gzip rely on statistical patterns. As an optimal prefix-free coding method, arithmetic coding's efficiency approaches the Shannon entropy limit but highly depends on accurate estimation of symbol probability distributions. Modern Transformer models learn long-range dependencies via self-attention mechanisms and can generate high-precision token conditional probability distributions. Combining the two is expected to break through the bottleneck of traditional compression ratios.

The system is divided into fine-tuning and inference phases. In the fine-tuning phase, the enwiki9 dataset is used to adapt the base model, with data tokenized into 64-token context-label pairs, supporting distributed training across 1-16 GPUs (PyTorch DDP). In the inference phase: After preprocessing, new text is input into the fine-tuned model to generate token probability distributions, which are converted into integer CDFs for the arithmetic encoder to produce bitstreams; decoding reversely reconstructs the token sequence.

The system supports four Transformer architectures: BERT (bidirectional encoder), RoBERTa (optimized BERT variant), T5-Small (encoder-decoder), and Llama-3.2-3B (open-source model with 3B parameters). It is optimized for the NVIDIA DGX A100 SuperPOD, which consists of 20 nodes (each with 8 A100 80GB GPUs), a total computing power of approximately 1.64 PFLOPS, 52.5TB of storage, and 200Gb/s InfiniBand inter-node connectivity.

The project establishes a comprehensive evaluation system: Compression performance metrics include compression ratio, BPC, BPT, cross-entropy, perplexity, KL divergence, and reconstruction accuracy; system performance metrics include wall-clock time, memory usage, and scaling efficiency. This framework provides data support for understanding the behavior of the hybrid pipeline in HPC environments and subsequent optimizations.

Open-source value: 1. First verification of the scalability of LLM compression on SOTA HPC platforms; 2. Systematic comparison of the performance of different Transformer variants; 3. Provision of complete SLURM scripts and distributed code. Limitations: High computational cost (fine-tuning Llama-3.2-3B requires significant computing power), higher latency than traditional algorithms (limiting application in real-time scenarios).

With improvements in model efficiency (quantization, pruning, distillation) and the popularization of AI acceleration hardware, the practicality of neural compression will improve. The LLM + arithmetic coding framework is an important exploration direction. The project code has been open-sourced (GitHub), including environment configuration, dataset guidelines, and multi-GPU configuration scripts.