HASTE: Accelerating Sparse Table Execution with High-Bandwidth Memory to Optimize Large Language Model Inference

The HASTE project explores how to accelerate sparse table execution using High-Bandwidth Memory (HBM), providing a new approach to performance optimization for Large Language Model (LLM) inference, aiming to address the efficiency bottleneck in LLM inference.

With the continuous expansion of Large Language Model (LLM) scales, inference efficiency has become a key bottleneck restricting their widespread application. Traditional dense computing modes face dual pressures of memory bandwidth and computing resources when handling large-scale parameters. As an effective optimization method, sparsification technology can significantly reduce computation and memory usage, but efficient execution of sparse operations remains an urgent technical challenge. Against this background, the HASTE project emerged to explore using HBM to accelerate sparse table execution for optimizing LLM inference.

Advantages of HBM

HBM achieves far higher bandwidth than traditional DDR memory through 3D stacking and wide bus architecture, which can effectively alleviate the memory bandwidth bottleneck in AI workloads.

Challenges in Sparse Table Execution

Sparse table execution involves a large number of random accesses to non-zero elements and irregular computations. Traditional dense matrix optimization techniques are difficult to apply directly, requiring specialized design of storage formats, index structures, and computation kernels.

HASTE's Innovative Ideas

• Efficient sparse data layout: Optimize the storage method of sparse tables in HBM to maximize access efficiency
• Parallel execution strategy: Design parallel computing modes suitable for HBM architecture
• Memory access optimization: Reduce performance loss caused by irregular access

Potential Impact on LLM Inference

1. Reduce inference latency: Accelerate sparse operations to shorten response time
2. Improve throughput: Process more requests per unit time
3. Reduce hardware costs: Use more cost-effective hardware for the same performance

Synergy with Existing Technologies

Can complement technologies such as quantization (INT8/INT4), pruning (structured/unstructured), speculative decoding, etc.

HASTE is an emerging open-source project currently in the early exploration stage, providing experimental reference implementations. In the future, we can expect to see more performance benchmark tests, optimization strategies, and sharing of practical deployment experiences.

HASTE represents an interesting exploration direction in the field of AI inference optimization. With the continuous growth of LLM scales, using hardware features (such as HBM) to accelerate sparse computing will become one of the key factors affecting model deployment efficiency, which is worthy of attention and follow-up by AI system optimization engineers and researchers.