Futhark Language Implements Qwen3 Inference: Functional GPU Programming Enters the LLM Inference Domain

The fuchat project uses the pure functional language Futhark to implement a complete inference engine for the Qwen3-0.6B model, demonstrating the potential of functional programming in GPU-accelerated LLM inference. Through KV caching and in-place update mechanisms, this implementation achieves a performance of 25 tokens/s, providing an innovative case for the application of functional languages in the LLM inference domain.

Optimization of large language model (LLM) inference has always been a core challenge in the AI engineering field. Traditionally, LLM inference frameworks mainly rely on C++, CUDA, or Python implementations, while functional programming languages are relatively rare in this domain. The emergence of the fuchat project breaks this pattern; it uses Futhark—a pure functional language designed for high-performance computing—to successfully implement a complete inference engine for the Qwen3-0.6B model.

Futhark is a programming language developed by the University of Copenhagen, focusing on compiling high-level functional code into efficient GPU kernels. Its unique features include support for nested parallelism and in-place array updates while maintaining pure functional semantics. This design philosophy gives it potential advantages in numerical computing and parallel processing tasks.

The fuchat project consists of two main components: the underlying Futhark inference engine and the upper-layer Python chat application. The inference engine implements key optimization techniques in modern LLM inference, including KV caching (Key-Value Cache) and prompt expansion mechanisms. KV caching significantly reduces the computational complexity of the self-attention mechanism by reusing previously computed key-value pairs during the decoding process.

The project uses the Qwen3-0.6B model by default, which is a lightweight version of Alibaba's Tongyi Qianwen series. Although the model size is small, the fuchat implementation demonstrates the feasibility of functional programming languages in handling complex neural network computations. On an AMD 6700XT graphics card (12GB VRAM), using Futhark's HIP backend, the f32 mode can achieve a generation speed of 20-25 tokens/s, and the f16 mode is about 10 tokens/s.

Performance data reveals some interesting phenomena. The f16 version of fuchat is actually about twice as slow as the f32 version, which is counterintuitive—usually half-precision computation should be faster. Developers speculate that this may be related to the level of optimization of the f16 type by the Futhark compiler, or changes in GPU memory access patterns.

More noteworthy is the performance improvement brought by KV caching. Before implementing KV caching, the pure f32 version had an inference speed of only 2-5 tokens/s. After introducing Futhark's "update in-place" mechanism, the performance improved by 5 to 10 times. This proves the effectiveness of the uniqueness typing system in functional languages when handling state-intensive computations.

For comparison, on the same hardware, llama.cpp can reach about 150 tokens/s using the f16 quantized model and about 110 tokens/s using the f32 quantized model. Fuchat still has a significant gap, but considering that this is a single-file, type-safe pure Futhark implementation, 25 tokens/s is already an impressive starting point.

The upper-layer Python chat application provides a complete interactive experience, supporting multi-turn conversations between users and the assistant role, a thinking mode switch (corresponding to Qwen3's reasoning ability), and simple Futhark entry point tool calls. This layered architecture separates performance-sensitive computation kernels from flexible application logic, which is a reasonable design choice.

The fuchat project raises a broader question: Can functional programming occupy a place in AI infrastructure? The traditional view is that the computational graph of neural networks is inherently stateful, conflicting with the immutable data model of functional programming. However, Futhark, through its unique in-place update semantics and parallel primitives, proves that functional abstractions and high-performance GPU computing can coexist.

For researchers and engineers who want to explore alternative implementation paths, fuchat provides a valuable reference point. It shows how to build an LLM inference system from first principles using a different approach than the mainstream technology stack.

To use fuchat, you need to install the nightly version of the Futhark compiler and configure a Python virtual environment. The project provides detailed compilation and running instructions. For developers interested in GPU programming languages, LLM inference optimization, or functional programming, this is an open-source project worth in-depth study.