TokenSpeed: A Blazing-Fast LLM Inference Engine for the Future

TokenSpeed is an LLM inference engine developed by the LightSeek team, positioned as a "speed-of-light LLM inference engine", currently in the preview phase. Its core goal is to achieve blazing-fast inference on next-generation hardware like NVIDIA B200, reproduce the inference performance of the Kimi K2.5 model, and demonstrate the effects of optimization technologies such as TokenSpeed MLA. This version is not recommended for production environments; it is mainly used to showcase the design of the next-generation runtime and technical directions, providing a reference implementation for researchers and developers.

With the rapid development of large language model technology today, inference efficiency has become a key bottleneck restricting the implementation of AI applications. As model scales continue to expand, how to achieve blazing-fast inference while ensuring output quality has become a core issue of industry concern. The TokenSpeed project was born to break through the performance limitations of traditional inference frameworks.

TokenSpeed's design revolves around three core goals: ultimate inference speed, full utilization of new hardware, and flexible model support. Its core features include:

1. Expanded Model Coverage: Integrates mainstream models such as Qwen 3.6, DeepSeek V4, and MiniMax M2.7, supporting both Chinese and international models;
2. Enhanced Runtime Functions: Developed speculative decoding (PD), expert parallel load balancing (EPLB), KV storage optimization, Mamba caching mechanism, VLM support, and performance monitoring;
3. Platform Optimization: Specialized optimizations for Hopper architecture (H100/H200) and AMD MI350.

TokenSpeed's core selling point lies in its deep optimization for new hardware. It has demonstrated impressive performance on NVIDIA B200 (the flagship product of the Blackwell architecture), and its design aims to fully exploit the memory bandwidth and computing power of this hardware. The project aims to reproduce the inference performance of Kimi K2.5 (a large multimodal model developed by MoonShot AI, known for its ultra-long context window and strong reasoning capabilities) on B200, which proves the forward-looking nature and depth of optimization of its architecture.

TokenSpeed provides a comprehensive documentation system to help developers get started quickly:

• Getting Started Guide: Quickly set up the environment and run inference tasks;
• Server Startup Documentation: Detailed steps for deploying inference services;
• Model Recipes: Optimization configuration recommendations for different models;
• Parameter Configuration Reference: Explanations of server and compatible parameters;
• Parallel Strategy Documentation: Parallel computing mechanisms and best practices.

Currently, TokenSpeed is in the preview phase and has several limitations: some core functions are still under development and not merged into the main branch; it is explicitly not recommended for production environments (due to insufficient stability, security, etc.). However, it has great potential for the future: as multimodal models, long-context inference, and real-time interactive applications become popular, the demand for inference speed is increasingly urgent, and TokenSpeed's "speed-of-light" inference concept may become a standard configuration for next-generation engines.

TokenSpeed represents a cutting-edge exploration in the field of LLM inference optimization. Although it is currently a preview version, its technical architecture, hardware adaptation strategy, and performance goals all show great potential. For researchers and developers focusing on AI infrastructure optimization, TokenSpeed is a project worth paying close attention to. With the improvement of functions and community contributions, it is expected to promote innovation in the inference efficiency of large models.