Runtime-Authenticated Bounded-Error Quantized Attention Mechanism

This paper proposes a hierarchical KV cache architecture to implement runtime-authenticated attention computation: INT8/INT4 quantized data resides in GPU memory to reduce footprint, while FP16 original values are retained in system memory for deterministic fallback; a two-term error decomposition provides per-head per-step error upper bounds; it matches FP16 quality on 128K context and can recover from catastrophic failures caused by naive quantization.

Long-context inference of large language models (LLMs) is limited by KV cache memory footprint. Quantization is a mainstream technique to reduce memory, but the introduced approximation errors pose reliability risks. Existing systems rely on the empirically verified assumption of 'average-case robustness' and cannot guarantee that specific inputs do not trigger catastrophic failures, which is unacceptable in high-risk scenarios (e.g., medical, legal).

The core idea is to transform quantization from 'fixed approximation' to 'runtime-verified computation': the system performs quantized attention while computing error upper bounds online, and automatically falls back to precise computation when the threshold is exceeded. Hierarchical architecture design: the GPU memory layer stores INT8 keys and INT4 values (high throughput, low footprint), and the system memory layer retains FP16 original key-value pairs (for error calculation and fallback).

The error is decomposed into two parts: 1) Attention distribution distortion caused by key quantization (upper bound depends on quantization step size, key vector norm, and dynamic range of attention scores); 2) Reconstruction error caused by value quantization (upper bound is related to quantization step size and statistical characteristics of value vectors, and is sensitive to task types). Bounds are computed in real-time during runtime with controllable overhead.

Adaptive precision selection: dynamically increase the value precision of a certain head when the error bound approaches the threshold; multi-level fallback ladder: first fall back value precision when exceeding the threshold, and if still not satisfied, fully fall back to FP16; deterministic guarantee: the output either meets the error bound or is equivalent to FP16 precise computation.

Tested on the LLaMA 3.1-8B model with 128K context, covering benchmarks like PG-19, NIAH, RULER: 1) Matches FP16 baseline quality; 2) Recovers from catastrophic failures caused by naive quantization; 3) Value-sensitive tasks can balance compression and fidelity by adjusting tolerance or fallback.

Insights: Average-case verification is insufficient; runtime authentication provides a security layer; quantization can use adaptive precision instead of all-or-nothing; hierarchical architecture suggests a direction for hardware-software co-design. Limitations: Authentication is local (per-head, per-step) not end-to-end; only for decoder-only architectures; system memory and GPU transfer overhead need to be balanced.