Intel Arc A770 LLM Inference Acceleration: SYCL Fused Kernel Achieves 40+ t/s Breakthrough

This project is maintained by hqh330 on GitHub (Project link: https://github.com/hqh330/arc770-llm, release date: 2026-05-23). Targeting the LLM inference performance bottleneck of llama.cpp on Intel Arc A770, it uses GPU-side dequantization and GEMM fusion technology to improve the inference speed of the Qwen2.5-7B Q4_K_M model from 26.4 t/s to over 40 t/s, achieving a 52% performance leap. The core optimizations are based on the SYCL fused kernel architecture and integration with IPEX-LLM.

In the field of LLM inference acceleration, NVIDIA GPUs have long dominated. Although the Intel Arc A770 has good hardware specifications (equipped with XMX units), the SYCL backend of llama.cpp lacks sufficient optimization, making it difficult to unleash its potential. In the traditional inference process, dequantization and GEMM are executed separately, leading to a large amount of data transfer between CPU and GPU, which forms a performance bottleneck. This project aims to address this pain point.

SYCL Fused Kernel Architecture

Embed dequantization operations inside the GEMM kernel to enable continuous execution on the GPU. The advantages include: reduced data transfer (quantized weights are directly transferred), elimination of synchronization overhead, improved cache efficiency, and full utilization of the potential of XMX units.

IPEX-LLM Integration

Integrate 48 SPIR-V kernel modules from Intel Extension for PyTorch (IPEX) LLM, optimized for the Arc architecture, supporting cross-GPU architecture porting and laying the foundation for future expansion.

Benchmark Test Results

Configuration	Inference Speed	Improvement
Baseline (unoptimized)	26.4 t/s	-
After fused kernel optimization	40+ t/s	+52%

Compilation Modes

• JIT Mode: Compilation takes 1-2 minutes, flexible for development and verification, but startup is slightly slow.
• AOT Mode: Compilation takes 15-20 minutes, precompiled into device binaries, fast startup and excellent performance, recommended for production deployment.

• DPAS Instruction Support: The branch includes support for the DPAS (Dot Product Accumulate Systolic) instruction, the core of the XMX unit, enabling efficient execution of matrix multiply-add operations.
• Memory Layout Optimization: Redesign the memory layout of Q4_K_M quantized weights, matching the fused kernel access pattern to achieve efficient vectorized loading.
• Multi-Queue Parallelism: The SYCL backend supports multi-queue parallelism, and task partitioning overlaps computation, data transfer, and preprocessing to improve throughput.

• Provide a practical performance improvement solution for Intel Arc users, solving the previously poor performance issue of llama.cpp.
• Demonstrate the value of the open-source community in filling gaps in vendor software ecosystems (deep integration of IPEX and llama.cpp relies on the community).
• Promote AI democratization: Lower the hardware threshold for running local LLMs, allowing more users to enjoy privacy and convenience.

• Expand support for more model architectures (Llama, Mistral, Phi, etc.).
• Add support for more quantization formats (Q5_K_M, Q6_K, etc.).
• Optimize small-batch inference latency.
• Explore multi-GPU parallel inference.
• Adapt to Intel Battlemage series graphics cards, expecting greater performance gains.