# Intel Arc A770 LLM Inference Acceleration: SYCL Fused Kernel Achieves 40+ t/s Breakthrough

> Using GPU-side dequantization and GEMM fusion technology, the inference performance of Qwen2.5-7B on Intel Arc A770 via llama.cpp is improved from 26.4 t/s to over 40 t/s, achieving a 52% performance leap.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-23T21:42:29.000Z
- 最近活动: 2026-05-23T21:49:54.725Z
- 热度: 154.9
- 关键词: Intel Arc A770, llama.cpp, SYCL, GPU加速, LLM推理, 量化模型, XMX, IPEX-LLM, Qwen2.5, 开源优化
- 页面链接: https://www.zingnex.cn/en/forum/thread/intel-arc-a770-llm-syclkernel40-t-s
- Canonical: https://www.zingnex.cn/forum/thread/intel-arc-a770-llm-syclkernel40-t-s
- Markdown 来源: floors_fallback

---

## Intel Arc A770 LLM Inference Acceleration: SYCL Fused Kernel Achieves 40+ t/s Breakthrough

This project is maintained by hqh330 on GitHub (Project link: https://github.com/hqh330/arc770-llm, release date: 2026-05-23). Targeting the LLM inference performance bottleneck of llama.cpp on Intel Arc A770, it uses GPU-side dequantization and GEMM fusion technology to improve the inference speed of the Qwen2.5-7B Q4_K_M model from 26.4 t/s to over 40 t/s, achieving a 52% performance leap. The core optimizations are based on the SYCL fused kernel architecture and integration with IPEX-LLM.

## Project Background and Motivation

In the field of LLM inference acceleration, NVIDIA GPUs have long dominated. Although the Intel Arc A770 has good hardware specifications (equipped with XMX units), the SYCL backend of llama.cpp lacks sufficient optimization, making it difficult to unleash its potential. In the traditional inference process, dequantization and GEMM are executed separately, leading to a large amount of data transfer between CPU and GPU, which forms a performance bottleneck. This project aims to address this pain point.

## Core Technical Solutions

### SYCL Fused Kernel Architecture
Embed dequantization operations inside the GEMM kernel to enable continuous execution on the GPU. The advantages include: reduced data transfer (quantized weights are directly transferred), elimination of synchronization overhead, improved cache efficiency, and full utilization of the potential of XMX units.
### IPEX-LLM Integration
Integrate 48 SPIR-V kernel modules from Intel Extension for PyTorch (IPEX) LLM, optimized for the Arc architecture, supporting cross-GPU architecture porting and laying the foundation for future expansion.

## Performance Test Results and Compilation Modes

#### Benchmark Test Results
| Configuration | Inference Speed | Improvement |
|------|---------|---------|
| Baseline (unoptimized) | 26.4 t/s | - |
| After fused kernel optimization | 40+ t/s | +52% |
#### Compilation Modes
- **JIT Mode**: Compilation takes 1-2 minutes, flexible for development and verification, but startup is slightly slow.
- **AOT Mode**: Compilation takes 15-20 minutes, precompiled into device binaries, fast startup and excellent performance, recommended for production deployment.

## Technical Implementation Details

- **DPAS Instruction Support**: The branch includes support for the DPAS (Dot Product Accumulate Systolic) instruction, the core of the XMX unit, enabling efficient execution of matrix multiply-add operations.
- **Memory Layout Optimization**: Redesign the memory layout of Q4_K_M quantized weights, matching the fused kernel access pattern to achieve efficient vectorized loading.
- **Multi-Queue Parallelism**: The SYCL backend supports multi-queue parallelism, and task partitioning overlaps computation, data transfer, and preprocessing to improve throughput.

## Application Value and Significance

- Provide a practical performance improvement solution for Intel Arc users, solving the previously poor performance issue of llama.cpp.
- Demonstrate the value of the open-source community in filling gaps in vendor software ecosystems (deep integration of IPEX and llama.cpp relies on the community).
- Promote AI democratization: Lower the hardware threshold for running local LLMs, allowing more users to enjoy privacy and convenience.

## Future Outlook

- Expand support for more model architectures (Llama, Mistral, Phi, etc.).
- Add support for more quantization formats (Q5_K_M, Q6_K, etc.).
- Optimize small-batch inference latency.
- Explore multi-GPU parallel inference.
- Adapt to Intel Battlemage series graphics cards, expecting greater performance gains.