SpikingLLM: Distribution-Aware Multi-Granularity Phase Encoding for Spiking-Driven Large Language Models

The SpikingLLM project proposes the distribution-aware multi-granularity phase encoding technology, which aims to solve conversion errors when combining spiking neural networks (SNNs) with large language models (LLMs) and achieve a high-performance, low-power neuromorphic computing architecture. This technology balances representational capability and computational efficiency through an adaptive encoding strategy, providing a new path for edge deployment, sustainable AI, and brain-inspired computing.

Large language models (LLMs) have strong intelligence but high energy consumption; the traditional Transformer architecture has high resource consumption for inference, limiting its application on edge devices. Spiking neural networks (SNNs) have event-driven characteristics that theoretically enable low power consumption, but combining them with LLMs faces accuracy loss—there is a fundamental difference between the discrete nature of spike activation and the continuous attention mechanism of Transformers.

Basics of Phase Encoding

Phase encoding simulates signal intensity through temporal position encoding of spike firing, making it more suitable for sequential tasks than rate encoding.

Multi-Granularity Strategy

Based on activation distribution differences across layers/channels, encoding precision is adaptively selected: fine granularity for high-information-density regions and coarse granularity for smooth regions, balancing representational capability and efficiency.

Distribution-Aware Mechanism

Dynamically monitor statistical characteristics of activation values (mean, variance, quantiles), adjust encoding parameters in real time, and reduce information loss during ANN-to-SNN conversion.

Spiking Neuron Layer

Uses the Leaky Integrate-and-Fire (LIF) neuron model to simulate membrane potential dynamics of biological neurons, enabling time-dimensional information accumulation and spike generation.

Attention Mechanism Transformation

Designs an approximate spiking attention unit to adapt to the SNN computing paradigm while maintaining self-attention expressive power.

Time Step Optimization

Compresses required time steps by optimizing encoding schemes and network structures, balancing energy efficiency and latency.

Evaluations on standard language modeling benchmarks show:

• Compared to baseline phase encoding, the distribution-aware multi-granularity strategy significantly reduces quantization error;
• Achieves an order-of-magnitude energy consumption reduction while maintaining similar model performance;
• Has good generalization across LLMs of different scales. These results provide empirical support for neuromorphic computing applications in LLMs.

Edge Deployment

Low-power characteristics allow LLMs to run locally on mobile phones and IoT devices, reducing cloud dependency and improving privacy and response speed.

Sustainable AI

Provides a path for green AI, reducing the carbon footprint caused by AI model expansion.

Brain-Inspired Computing

Deepens understanding of information processing mechanisms in biological nervous systems, laying the foundation for building brain-like efficient AI systems.

Current limitations: Mainly optimizes forward inference energy efficiency; spike learning algorithms in the training phase need improvement. Future directions: Expand to multimodal large models and complex reasoning tasks; deeply integrate with neuromorphic hardware (e.g., Intel Loihi, IBM TrueNorth) to unleash SNN potential.