Reading
FlashRT is a high-performance real-time inference engine designed specifically for small-batch, latency-sensitive AI workloads. It supports VLA robot control models and LLM inference, achieving extremely low latency through handwritten CUDA kernels and static graph capture.
Section 01
FlashRT is a high-performance real-time inference engine designed specifically for small-batch, latency-sensitive AI workloads. It supports VLA robot control models and LLM inference, achieving extremely low latency through handwritten CUDA kernels and static graph capture.
Section 02
Section 03
Current mainstream AI inference tools like TensorRT and vLLM are designed for different workload scenarios: TensorRT focuses on compiling models into frozen engines to support tactical search, while vLLM targets large-batch LLM services. However, for small-batch real-time inference scenarios—especially robot Vision-Language-Action (VLA) models and real-time LLM services—existing inference frameworks often face issues such as high compilation overhead, high startup latency, and difficulty adapting quickly to model changes.
FlashRT emerged to fill this gap. It is specifically designed for small-batch, latency-sensitive real-time inference scenarios, enabling a compile-free, plug-and-play inference experience through handwritten CUDA kernels and static graph capture technology.
Section 04
The core advantage of FlashRT lies in its handwritten CUDA kernels, which cover all memory-bound operations required by standard Transformer, DiT, and SigLIP primitives:
These kernels are specifically optimized for memory bandwidth bottlenecks, rather than compute-intensive operations (GEMM/attention), which are delegated to cuBLASLt, CUTLASS, and vendor FA2 implementations.
Section 05
FlashRT captures the entire forward propagation process as a static CUDA graph, enabling graph replay with zero Python overhead. This design brings significant advantages:
Section 06
FlashRT automatically detects GPU capabilities during load_model() and routes to the best-matching backend:
| Compute Capability | GPU | Backend |
|---|---|---|
| SM110 (11.0) | Jetson AGX Thor | flash_rt.hardware.thor.* |
| SM120 (12.0) | RTX 5090 Blackwell | flash_rt.hardware.rtx.* |
| SM89 (8.9) | RTX 4090 Ada | flash_rt.hardware.rtx.* |
User code does not need to be aware of the underlying hardware; the same code can run seamlessly on Jetson Thor, RTX 5090, and RTX 4090.
Section 07
FlashRT's flagship integration is production-grade VLA control, supporting the following models:
| Model | Thor FP8 Latency | RTX 5090 FP8 Latency |
|---|---|---|
| Pi0.5 | 44.0 ms (23 Hz) | 17.58 ms (57 Hz) |
| Pi0 | 45.8 ms (22 Hz) | 18.43 ms (54 Hz) |
| GROOT N1.6 | 41 ms (24 Hz) | 12.53 ms (80 Hz) |
| Pi0-FAST | 8.1 ms/token | 2.39 ms/token |
Section 08
FlashRT also supports pure-text LLM inference:
Keep going with more reads from the same topic.
Nornir MCP Server is an enterprise-level server based on the Model Context Protocol (MCP). It seamlessly integrates large language models (such as Claude) with the Nornir network automation framework, supporting natural language orchestration for multi-vendor network devices (Cisco, Arista, Juniper, etc.), and providing production-grade features like a dual-engine architecture (NAPALM + Netmiko), intelligent filtering, and a secure sandbox.
Bibliothèque Française LLM is a structured indexing and annotation project for French public domain literature designed specifically for large language models (LLMs). It integrates multiple authoritative sources such as DraCor, Common Corpus, and Wikisource, providing metadata indexing categorized by genre, author, and era, as well as in-depth annotations for dramatic texts (including characters, lines, stage directions, etc.). Its aim is to enable LLMs to efficiently read and understand classic French literary works.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
libmlxforge is an embeddable MLX large language model (LLM) inference engine designed specifically for Apple Silicon. It provides a unified C ABI interface, supports calls from Node.js, Swift, and Rust, and features continuous batching, streaming output, JSON-constrained structured output, and embedding vector generation.