IBP: A New Algorithm to Break GPU Memory Bottlenecks via Lossless Compression

Invariant Bit Packing (IBP), a new lossless compression algorithm designed specifically for machine learning workloads, can significantly improve the performance of GNN training, recommendation systems (DLRM), and LLM inference without losing precision, effectively breaking GPU memory bottlenecks. This research comes from an arXiv paper (published on May 29, 2026, link: http://arxiv.org/abs/2605.30728v1).

In machine learning training and inference, dataset sizes often exceed GPU memory capacity, requiring tensor transfer via PCIe, which becomes a performance bottleneck. Existing lossy compression solutions have precision loss, complex deployment, or are even unacceptable; while lossless compression can preserve data integrity, the key lies in how to integrate it into ML pipelines with minimal GPU interference.

IBP achieves lossless compression by identifying and eliminating invariant bits in tensor groups: 1. Invariant bit identification: Analyze data patterns in tensor groups to find invariant bits within the group; 2. Bit packing: Eliminate redundant invariant bits and retain only the varying parts; 3. GPU-optimized decompression: Use warp parallelism, low-overhead bit operations, and asynchronous PCIe transfers for efficient decompression. Its features include losslessness, high throughput, low latency, and versatility (easy-to-use API for integration into multiple ML frameworks).

IBP performs significantly in representative ML workloads: GNN training is accelerated by an average of 74% (reducing CPU-GPU data transfer); DLRM embedding lookup is accelerated by an average of 180% (optimizing access to large embedding tables); LLM inference is accelerated by an average of 24% (still a considerable improvement in highly optimized scenarios).

The research team provides an easy-to-use API that can be integrated into GNN training frameworks, DLRM, and LLM inference frameworks; IBP is designed with compatibility for mainstream ML frameworks in mind, requiring no modification to model architectures or training algorithms, and its "plug-and-play" feature lowers the adoption barrier.

IBP has important implications in multiple scenarios: Cloud ML services can reduce costs (acceleration translates to resource savings); edge devices can run larger models to expand AI deployment; large-scale training reduces communication overhead and improves scaling efficiency.

IBP has limitations: Compression effectiveness depends on data structure (poor compression ratio for highly random data); current optimizations are for specific GPU architectures, and effects on other hardware need to be verified; future directions can explore hybrid strategies of IBP and lossy compression.

IBP demonstrates the feasibility and effectiveness of lossless compression in ML workloads, providing a new path to break GPU memory bottlenecks without precision loss. For ML engineers and researchers facing memory bottlenecks, IBP is a worthy optimization option to consider. The extended version of the paper contains more details; you can visit arXiv to get the full content.