Imp: A High-Performance LLM Inference Engine Built for NVIDIA Blackwell Architecture

Imp is a high-performance LLM inference engine developed with C++/CUDA, specifically optimized for NVIDIA's new Blackwell architecture GPUs (e.g., RTX 5090) to fully unleash the computing potential of next-gen hardware. This thread covers its background, core technical features, performance benchmarks, application scenarios, and future plans.

Project Background

LLM inference efficiency is a bottleneck for large-scale applications. As model parameters grow to hundreds of billions, hardware demands rise. NVIDIA's 2025 Blackwell architecture brings unprecedented computing power and AI acceleration, but existing engines (for Ampere/Hopper) can't utilize its new features—leading to Imp's creation.

Blackwell's Key Innovations

1. 5th Gen Tensor Core: Supports FP8/FP6 with micro-tensor scaling for better throughput and stability.
2. Decompression Engine: Real-time decompression during memory transfer boosts effective bandwidth, critical for autoregressive tasks.
3. Multi-GPU Upgrade: Enhanced NVLink/NVSwitch for higher bandwidth/lower latency, enabling efficient distributed inference for long contexts and multi-modal apps.

Native Blackwell Optimization

• FP8 Support: Full FP8 compute (forward/backward) with fine scaling to maintain FP16-level precision.
• Asynchronous Pipeline: Orchestrates compute, memory transfer, and communication to minimize idle time.
• Dynamic Batching: Auto-adjusts batch size based on load to balance latency and throughput.

Memory Efficiency

• Quantization: Supports INT8/FP8/mixed precision for flexible tradeoffs.
• PagedAttention: Manages KV cache as non-contiguous blocks to reduce fragmentation.
• Weight Sharing: Cross-instance weight reuse for multi-instance deployments.

High-Performance Kernels

• FlashAttention-3 Variant: Optimized for Blackwell's memory access and parallelism.
• Custom GEMM: Specialized for LLM's long-narrow matrices, 30% faster than cuBLAS in some cases.
• Operator Fusion: Merges small ops to cut kernel overhead and memory round trips.

Single-Card Performance

On RTX5090, Imp outperforms vLLM on Llama-3-70B: +25% throughput, -15% first-token delay (due to Blackwell feature utilization).

Multi-Card Scalability

8-card setup achieves near-linear scaling efficiency, ideal for ultra-large models (e.g., GPT-4 level).

Energy Efficiency

20% higher task per unit power than competitors, reducing data center operational costs.

Production Services

Offers monitoring, health checks, fault recovery, and OpenAI-compatible API for easy integration.

Local Development

Flexible configs and debug tools for researchers to test optimization strategies.

Edge Deployment

Modular design supports移植 to Blackwell-based Jetson devices for edge AI applications.

Ecosystem

• vs vLLM: Complementary—vLLM for broad compatibility, Imp for Blackwell's ultimate optimization.
• vs TensorRT-LLM: More open/agile, allowing faster community iteration.

Technical Challenges & Solutions

• Compile Complexity: Auto-tuning system selects optimal kernel configs for hardware/workload.
• Precision-Efficiency Tradeoff: Dynamic precision adjusts based on input complexity.
• Long Context: Improved KV cache management + sparse attention for million-token contexts.

Future Roadmap

• Multi-modal support (vision-language models, cross-modal attention).
• Speculative decoding to reduce generation latency.
• Enhanced distributed inference for larger models.

Conclusion

Imp marks a new era of hardware-specialized LLM inference. It provides an option for users seeking ultimate performance and offers valuable open-source references for the community. As AI chips evolve, more such specialized engines are expected.