Tattletale: A High-Performance Multimodal LLM Inference Engine with Cross-Platform Support for CUDA, Vulkan, and WebGPU

Tattletale is a high-performance multimodal LLM inference engine developed in Nim, aiming to break the contradiction between performance and portability in the field of large language model inference. It supports multiple backends such as CUDA, OpenCL, Vulkan, and WebGPU, and features an innovative IntrusiveAttention cache mechanism, EXL3 quantization support, and Lean4 formal verification. Its goal is to achieve both high-performance inference and true cross-platform compatibility.

Original Author and Source

• Original Author/Maintainer: mratsim
• Source Platform: GitHub
• Original Link: https://github.com/mratsim/tattletale
• Release Date: 2026-06-02

Project Background

In the field of large language model inference, performance and portability often conflict: most engines either focus on a single platform for extreme performance or sacrifice performance for cross-platform compatibility. Tattletale attempts to break this dilemma through an innovative architecture.

Key Technologies

1. IntrusiveAttention Cache Mechanism: A PagedRadixTrie implemented based on an intrusive WAVL tree, optimizing KV cache management.
2. Nim-to-GPU Compiler: Generates multi-backend (CUDA/OpenCL/Vulkan/WebGPU) code via Nim macros to enable cross-platform support.
3. EXL3 Quantization Scheme: Uses techniques like random Hadamard rotation, Trellis quantization, and lattice codebooks to balance model size and performance.

Architectural Design Principles

• Embedded and minimal dependencies: Currently only depends on drivers and libTorch C++, with plans for zero dependencies in the future.
• Portable code generation: Generates platform-optimized code at build/runtime.
• Formal verification: Uses Lean4 to verify complex state management logic.

Advantages of IntrusiveAttention

• Worst-case latency guarantee: Avoids hash table reconstruction/tombstone issues, ensuring stable performance based on WAVL tree characteristics.
• Efficient prefix matching: Approximately 50ns + O(memory bandwidth) complexity, capable of handling over 100,000 cache requests per machine.
• Formal verification: Core logic has been verified with Lean4 (code links: Implementation, Verification)

Nim Compiler Implementation

Compiler code is located at: https://github.com/mratsim/tattletale/tree/dbb44dd/workspace/positron/src/codegen

Technology Stack Status

Component	Current Status	Future Plan
GPU Backend	CUDA, OpenCL, Vulkan, WebGPU	Add HIP, Metal, DX12
Tensor Library	libTorch C++	Self-developed tensor library
KV Cache	IntrusiveAttention	Continuous optimization
Quantization	EXL3	Support more schemes
Verification	Partially verified with Lean4	Expand scope
Modalities	Text	Audio, Image

Applicable Scenarios

1. Cross-platform AI applications: Simplifies development and maintenance for desktop, mobile, and web platforms.
2. High-concurrency inference services: IntrusiveAttention supports efficient concurrent queries.
3. Edge device deployment: Quantization and multi-backend adaptation for resource-constrained devices.
4. Browser-side inference: WebGPU enables running large models directly in browsers without a backend.

Ongoing Work

• Porting core ideas from CuteDSL/Cutlass/TileLang to Nim to enhance GPU kernel generation capabilities.

Future Plans

• Completely remove libTorch dependency and develop a self-owned tensor library (the author has relevant experience with Arraymancer, etc.).

Community Participation

The project is in the early stage. The GitHub repository provides explanations of motivation and MVP goals: https://github.com/mratsim/tattletale/issues/1. Developers interested in high-performance inference, cross-platform GPU programming, and formal verification are welcome to follow.

Tattletale is an LLM inference engine project that combines technical ambition and innovation. Through innovations like IntrusiveAttention and the Nim-to-GPU compiler, combined with Lean4 formal verification, it brings new possibilities to the field of LLM inference. For developers building high-performance, cross-platform AI applications, Tattletale is a direction worth paying attention to, and it is expected to become an important choice in this field in the future.