VitaLLM: An Ultra-Compact Ternary LLM Accelerator for Edge Devices

Introduction: VitaLLM is a hardware-software co-designed ternary LLM inference accelerator for edge devices. Through innovations like the heterogeneous dual-core computing strategy and dependency-aware scheduling framework, it achieves a decoding throughput of 70.70 tokens/s with an area of 0.223 mm² and power consumption of 65.97 mW, providing an efficient solution for edge LLM deployment.

Background: Deploying large language models (LLMs) on edge devices faces two core obstacles—memory bandwidth bottleneck (frequent access to parameters and KV cache during inference leads to idle computing units) and power constraints (high energy consumption of traditional high-precision operations). Ternary quantization (e.g., BitNet b1.58) can compress the model to 1/16 of its original size while maintaining accuracy, but general-purpose hardware deployment has issues like workload imbalance, decoding bandwidth bottlenecks, and data dependencies.

Method: VitaLLM adopts a heterogeneous dual-core computing strategy, with division of labor for different tasks:

• TINT-Cores: Optimized for projection operations in ternary matrix multiplication, efficiently executing dot product calculations of {-1,0,+1};
• BoothFlex-Core: An attention core supporting mixed-precision operations, using improved Booth encoding to handle attention mechanism requirements;
• Collaboration mechanism: TINT-Cores are used for parallel computing in the pre-filling phase, while BoothFlex-Core handles attention in the decoding phase, improving utilization in each phase.

Method: VitaLLM introduces two major optimization mechanisms:

1. Leading-One Prediction (LOP) Mechanism: By predicting the distribution of attention scores, it prunes redundant KV cache reads to reduce memory access;
2. Dependency-Aware Scheduling Framework: Analyzes computational graph dependencies, builds fine-grained pipelines, and hides the latency of non-linear operations (activation, normalization) through prefetching and speculative execution.

Evidence: VitaLLM is implemented based on TSMC 16nm process, with key indicators:

• Decoding throughput: 70.70 tokens/s;
• Chip area: 0.223 mm²;
• Power consumption: 65.97 mW;
• Performance density: 17.4 TOPS/mm²/W (Figure of Merit). Compared to existing advanced accelerators, its performance density is significantly improved. 70.70 tokens/s supports smooth dialogue, and its low power consumption and small area make it suitable for edge device integration.

Extension: The research team explored the bit-serial design extension BoothFlex-BS:

• Precision agility: Dynamically adjusts computing precision at runtime to achieve precision-efficiency trade-off (low precision for throughput, high precision for quality);
• Architecture adaptability: Verifies the scalability of the VitaLLM architecture, which can adapt to different application requirements.

Impact: VitaLLM breaks down barriers to edge LLM deployment:

• Privacy protection: Local inference avoids uploading data to the cloud, reducing privacy risks in sensitive scenarios (medical, financial);
• Offline availability: Provides AI services even in no-network or weak-network environments, suitable for remote areas and emergency scenarios;
• Cost-effectiveness: Reduces cloud dependency and lowers enterprise operating costs;
• Widespread device integration: Small area and low power consumption enable integration into mobile phones, IoT devices, and wearables.

Outlook and Conclusion: VitaLLM represents an important direction for edge AI accelerators:

• Deep integration of quantization and dedicated hardware: Extreme quantization (binary, ternary) and hardware co-design have great potential;
• Dynamic precision adjustment: Optimize efficiency on demand;
• Memory-computing integration: Reduce data movement overhead. VitaLLM proves the feasibility of running LLMs on edge devices, promoting the vision of "AI everywhere". In the future, more efficient intelligent services will be deployed on edge devices.