SpikingLLM: Reducing Conversion Error of Spiking-Driven Large Language Models via Distribution-Aware Multi-Granularity Phase Coding

Open-source implementation of an ICLR 2026 accepted paper, proposing a distribution-aware multi-granularity phase coding method that effectively reduces ANN-to-SNN conversion error, enabling efficient spiking neural network inference on LLaMA-2 and LLaMA-3 models.

• Original Author/Maintainer: njzhenghy
• Source Platform: GitHub
• Original Title: SpikingLLM
• Original Link: https://github.com/njzhenghy/SpikingLLM
• Source Publication/Update Time: 2026-06-16T09:13:56Z

Spiking Neural Networks (SNNs), known as the third generation of neural networks, have attracted much attention due to their event-driven computing characteristics and biological interpretability.

Compared to traditional Artificial Neural Networks (ANNs), SNNs consume energy only when neurons fire spikes; this sparse activation feature gives them a huge advantage in energy efficiency, making them particularly suitable for edge computing and neuromorphic chip deployment.

However, applying SNNs to the field of Large Language Models (LLMs) faces severe challenges. Due to the fundamental difference between the discrete spike mechanism of SNNs and the continuous activation functions of LLMs, directly converting pre-trained LLMs to SNNs results in significant accuracy loss, a problem known as the 'ANN-to-SNN conversion error'. Existing conversion methods often struggle to achieve efficient spike inference while maintaining model performance.

The research team led by NJ Zheng et al. addressed this problem by proposing the 'Distribution-Aware Multi-Granularity Phase Coding' method, which successfully enables efficient spike-driven inference for LLaMA series models, and the related results have been accepted by ICLR 2026.

Phase Coding is an important temporal coding method in SNNs, which uses the timing of spike firing to encode information.

Compared to traditional Rate Coding, Phase Coding can transmit more information in fewer time steps, thereby improving the inference efficiency of SNNs.

In Phase Coding, the activation value of a neuron is encoded as the firing time of the spike within a specific time window. For example, a higher activation value corresponds to an earlier spike firing time, while a lower activation value corresponds to a later firing time. This coding method allows SNNs to transmit analog value information in a single time step, greatly improving information transmission efficiency.

The research team found that single-granularity phase coding is difficult to adapt to the differences in activation distributions of different layers and neurons in LLMs.

To address this, they proposed the 'Multi-Granularity Phase Coding' strategy, which allows the model to adaptively select the coding granularity based on the distribution characteristics of activation values.

Specifically, this method groups neurons, and each group uses a different coding granularity (grain). For example, some groups may use 2-level granularity (dividing the activation range into 2 intervals), while others may use 3-level granularity (dividing the activation range into 3 intervals). This flexible grouping strategy allows the coding to better match the actual activation distribution of each neuron group.

'Distribution Awareness' is one of the core innovations of this method.

The research team analyzed the statistical distribution of activation values in each layer of LLMs and identified that neurons in different layers and positions have different activation distribution characteristics. Based on this distribution information, they designed an optimization algorithm that automatically selects the most appropriate coding granularity for each neuron group.

This distribution-aware method ensures that coding resources are reasonably allocated: for neuron groups with a relatively concentrated activation distribution, using a coarser granularity is sufficient to ensure accuracy; for neuron groups with a relatively dispersed activation distribution, a finer granularity is needed to fully express the information.

This project provides complete training and conversion code, supporting ANN-to-SNN conversion for LLaMA-2-7B and LLaMA-3-8B models. Experimental results show that this method achieves excellent performance on multiple benchmark tests:

LLaMA-2-7B Experimental Results (using 8 time steps, T=8):

• WikiText-2 Perplexity: 5.50 (grain=2) / 5.50 (grain=3)
• WinoGrande Accuracy: 70.48%
• ARC-Challenge Accuracy: 46.50% (grain=2) / 46.33% (grain=3)
• ARC-Easy Accuracy: 73.91% (grain=2) / 73.86% (grain=3)
• PIQA Accuracy: 78.29% (grain=2) / 78.35% (grain=3)

LLaMA-3-8B Experimental Results (using 8 time steps, T=8):

• WikiText-2 Perplexity: 6.34 (grain=2) / 6.33 (grain=3)
• WinoGrande Accuracy: 72.93% (grain=2) / 73.72% (grain=3)
• ARC-Challenge Accuracy: 54.01% (grain=2) / 53.41% (grain=3)
• ARC-Easy Accuracy: 77.44% (grain=2) / 77.36% (grain=3)
• PIQA Accuracy: 80.63% (grain=2) / 80.36% (grain=3)

These results indicate that even with a small number of time steps (e.g., 6-10 steps), this method can still maintain high model performance, significantly outperforming traditional ANN-to-SNN conversion methods.

Fast Hadamard Transform: The project uses the fast-hadamard-transform library developed by Dao-AILab for efficient computation of Hadamard transforms, which is a key mathematical tool for implementing phase coding.

Grain Analysis Optimization: The research team uses the Grain Analysis module to analyze neuron activation distributions and select the optimal coding granularity for each neuron group. The optimized parameter configuration has further improved the results compared to those reported in the original paper.

Training Framework: The project is built based on PyTorch 2.4.1, supports CUDA 12.4, and integrates efficient attention mechanism implementations such as Flash Attention.