Running a 200M-Parameter Large Model on Mobile: Butterfly Transform Enables On-Device AI Training

An open-source project named "on-device-butterfly-llm" enables direct training of a 200M-parameter large language model on Android phones without cloud support. Using the Diagonal-Interleaved Butterfly (DIB) attention mechanism and NEON SIMD optimization, it achieves 10x faster inference speed than traditional methods and reduces memory usage by over 50x, opening up new possibilities for privacy-sensitive applications and offline scenarios.

Large language model training and inference have long relied on cloud GPU clusters, with exponentially increasing computing power and memory demands leading to high costs. This project completed training in the Termux environment on a Poco F5 phone equipped with the Snapdragon 7+ Gen 2 processor, subverting the perception that large models cannot be trained on devices, proving technical feasibility, and providing solutions for privacy and offline scenarios.

The traditional transformer attention mechanism has a complexity of O(N²), with memory expanding sharply as sequence length increases. DIB attention reduces complexity to O(N log N) by decomposing the N×N matrix into butterfly transformations plus diagonal gating. Tests show: for a matrix dimension of 2048, the traditional method requires 16MB of memory, while the Butterfly version only uses 0.172MB (93x compression ratio); for dimension 8192, the compression ratio reaches 315x, enabling deployment of larger models.

Deeply optimized for ARM architecture, using the NEON SIMD instruction set (dotprod and fp16 extensions of ARMv8.4-a) and OpenMP multi-threading, it achieves an inference speed of 5263 tokens/s on the Snapdragon 7+ Gen2 (10x faster than unoptimized). A 10-second continuous test showed no thermal throttling, with a stable temperature of 37.9°C and proper power consumption control.

The project implements Flash-LoRA fine-tuning, integrating LoRA weights into attention head computation with an additional overhead of only 2.41%. Predictive coding solves the gradient vanishing problem, improving error decay from O(exp(-L)) to O(1). In tests, LoRA fine-tuning consumed 0.0287 joules, which is 697x more efficient than cloud training (20 joules), making it suitable for mobile devices.

Potential applications include local processing of privacy-sensitive data, offline use, and reducing AI deployment costs. The project's next steps are to expand the model scale and organize results for submission to arXiv; currently, the paper's publishability score is 8.2/13.0, gaining academic recognition.

Through three breakthroughs—algorithm innovation, architecture optimization, and hardware adaptation—this project enables cloud-level tasks on ordinary phones, promoting AI democratization and privacy protection. As on-device chip performance improves and optimization technologies mature, more powerful AI capabilities will run on mobile devices in the future.