# FlashRT: A High-Performance Inference Engine for Real-Time AI Workloads

> FlashRT is a high-performance real-time inference engine designed specifically for small-batch, latency-sensitive AI workloads. It supports VLA robot control models and LLM inference, achieving extremely low latency through handwritten CUDA kernels and static graph capture.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-06-19T17:08:59.000Z
- 最近活动: 2026-06-19T17:23:34.418Z
- 热度: 173.8
- 关键词: FlashRT, CUDA, 实时推理, VLA, 机器人控制, LLM推理, 静态图, 边缘计算, Jetson, RTX 5090, Pi0, GROOT, 量化推理, FP8, NVFP4
- 页面链接: https://www.zingnex.cn/en/forum/thread/flashrt-ai-6e041de8
- Canonical: https://www.zingnex.cn/forum/thread/flashrt-ai-6e041de8
- Markdown 来源: floors_fallback

---

## Introduction / Main Post: FlashRT: A High-Performance Inference Engine for Real-Time AI Workloads

FlashRT is a high-performance real-time inference engine designed specifically for small-batch, latency-sensitive AI workloads. It supports VLA robot control models and LLM inference, achieving extremely low latency through handwritten CUDA kernels and static graph capture.

## Original Author and Source

- **Original Author/Maintainer**: flashrt-project organization
- **Source Platform**: GitHub
- **Original Title**: FlashRT
- **Original Link**: https://github.com/flashrt-project/FlashRT
- **Publication Date**: June 19, 2026

## Background and Challenges

Current mainstream AI inference tools like TensorRT and vLLM are designed for different workload scenarios: TensorRT focuses on compiling models into frozen engines to support tactical search, while vLLM targets large-batch LLM services. However, for small-batch real-time inference scenarios—especially robot Vision-Language-Action (VLA) models and real-time LLM services—existing inference frameworks often face issues such as high compilation overhead, high startup latency, and difficulty adapting quickly to model changes.

FlashRT emerged to fill this gap. It is specifically designed for small-batch, latency-sensitive real-time inference scenarios, enabling a compile-free, plug-and-play inference experience through handwritten CUDA kernels and static graph capture technology.

## 1. Handwritten CUDA Kernel Library

The core advantage of FlashRT lies in its handwritten CUDA kernels, which cover all memory-bound operations required by standard Transformer, DiT, and SigLIP primitives:

- **Normalization and activation layers**: LayerNorm, RMSNorm, SwiGLU, etc.
- **Fused operators**: Residual + normalization + quantization fusion, RoPE/qkv-split
- **Quantization support**: FP8, NVFP4 GEMM, cuBLASLt FP8
- **Attention mechanisms**: Vendor-optimized Flash-Attention 2, Thor CUTLASS FMHA

These kernels are specifically optimized for memory bandwidth bottlenecks, rather than compute-intensive operations (GEMM/attention), which are delegated to cuBLASLt, CUTLASS, and vendor FA2 implementations.

## 2. Static CUDA Graph Capture

FlashRT captures the entire forward propagation process as a static CUDA graph, enabling graph replay with zero Python overhead. This design brings significant advantages:

- **First call**: ~3 seconds (calibration + CUDA graph capture)
- **Subsequent calls**: Only 44ms graph replay (Thor platform)
- **No engine files**: No need to export ONNX or compile engines
- **Driver upgrade immunity**: No need to rebuild when CUDA driver is upgraded, GPU is replaced, or prompt changes

## 3. Automatic Hardware Distribution

FlashRT automatically detects GPU capabilities during `load_model()` and routes to the best-matching backend:

| Compute Capability | GPU | Backend |
|---------|-----|------|
| SM110 (11.0) | Jetson AGX Thor | `flash_rt.hardware.thor.*` |
| SM120 (12.0) | RTX 5090 Blackwell | `flash_rt.hardware.rtx.*` |
| SM89 (8.9) | RTX 4090 Ada | `flash_rt.hardware.rtx.*` |

User code does not need to be aware of the underlying hardware; the same code can run seamlessly on Jetson Thor, RTX 5090, and RTX 4090.

## VLA Model Support

FlashRT's flagship integration is production-grade VLA control, supporting the following models:

| Model | Thor FP8 Latency | RTX 5090 FP8 Latency |
|------|-------------|------------------|
| Pi0.5 | 44.0 ms (23 Hz) | 17.58 ms (57 Hz) |
| Pi0 | 45.8 ms (22 Hz) | 18.43 ms (54 Hz) |
| GROOT N1.6 | 41 ms (24 Hz) | 12.53 ms (80 Hz) |
| Pi0-FAST | 8.1 ms/token | 2.39 ms/token |

## LLM Support

FlashRT also supports pure-text LLM inference:

- **Qwen3.6-27B NVFP4**: Supports 256K context on RTX 5090, 145 tok/s warm-up decoding
- **Qwen3-8B NVFP4**: 9.1ms prefill at P=64, ~150 tok/s decoding