Planckify: An Experimental Project for Edge Large Model Inference Based on Google LiteRT-LM

Planckify is an open-source experimental project focused on edge large language model inference. Using the Google LiteRT-LM framework and starting with the Gemma 4 E2B model, it explores the feasibility of running large language models in a pure CPU environment. This project aims to solve issues like latency, privacy concerns, and network dependency in cloud-based inference, promoting the implementation of edge AI technology.

With the development of LLM technology, edge inference has become a popular direction. Cloud-based inference has issues like high latency, privacy risks, and network dependency, which edge inference can solve by running models locally. In recent years, advances in model compression, quantization technologies, and dedicated frameworks (such as Google LiteRT-LM) have made it possible for consumer-grade hardware to run models with billions of parameters.

Planckify is an open-source experimental project that selects Google LiteRT-LM as its underlying framework and starts with the Gemma 4 E2B model. Gemma is a lightweight open-source model by Google; the 4B version is small in size and has good language capabilities, while the E2B version is optimized to be more suitable for edge devices.

LiteRT-LM Framework

LiteRT-LM is optimized for mobile/edge devices, with advantages including a lightweight runtime, cross-platform support, hardware acceleration, and quantization support (INT8/INT4).

CPU Inference Challenges and Optimization

Challenges: Memory bandwidth bottlenecks, low efficiency of compute-intensive operations. Optimization Strategies:

• Memory Optimization: Memory management to reduce allocation and copying, memory-mapped loading of weights
• Compute Optimization: SIMD instruction set to accelerate matrix operations, block computation to improve cache hit rate
• Quantization Inference: Converting FP32 to INT8/INT4 to reduce memory usage and bandwidth requirements

Planckify successfully runs the Gemma 4 E2B model in a CPU environment. The performance focus dimensions include:

• Inference Latency: First token generation time, subsequent token speed
• Memory Usage: Peak memory consumption
• Model Quality: Impact of quantization on output quality (perplexity, task accuracy)
• Energy Efficiency: Inference energy consumption on battery-powered devices

Edge LLM inference can enable multiple scenarios:

• Privacy-sensitive applications: Local processing of medical/financial data to protect privacy
• Offline availability: Usable in network-free environments (airplanes/remote areas)
• Low-latency interaction: Real-time voice assistants, translation, etc.
• Personalized models: Local fine-tuning to create personalized AI assistants

Challenges:

• Trade-off between model size and capability: Edge models (e.g., 4B parameters) are less capable of complex tasks than cloud-based large models
• Heterogeneous computing optimization: Efficient use of heterogeneous resources like GPU/NPU
• Dynamic loading and unloading: Dynamic management of layers in ultra-large models
• Development toolchain: Need to improve tools for model conversion, quantization, and performance analysis Future Directions: Continuously optimize the balance between resources and capabilities, and improve the toolchain

Planckify has verified the feasibility of running the Gemma 4B model on edge CPUs, which is a beneficial exploration of edge LLM inference. With hardware advancements and software optimizations, more AI capabilities will run locally on daily devices in the future. Developers can enter the edge AI field through the LiteRT-LM framework, Gemma models, and the Planckify open-source project.