Research on Activation Value Measurement of Open-Source Large Models: Revealing Hidden Risks in Quantization Deployment

This article conducts a systematic measurement study on the dynamic range of activation values in modern open-source large language models. It finds that the maximum activation values of different model families differ by nearly four orders of magnitude, activation values of MoE architectures are significantly lower than those of Dense models of the same scale, and residual streams carry the global maximum activation values. These findings have important guiding significance for low-bit quantization deployment, emphasizing that activation values should be measured and reported as model attributes.

In the actual deployment of large language models, the dynamic range of activation values directly affects low-bit quantization, scaling, and inference stability. Early studies were based on LLaMA series before 2024 and did not verify the rules of new architectures such as Qwen, Gemma, and Mixtral; existing quantization toolchains rely on early conclusions, which may lead to deployment issues. Core research questions: The magnitude of activation values of modern open-source models and the differences between different model families/generations/training stages.

Dataset and Preprocessing

A multi-domain corpus of 5000 samples (covering news, encyclopedias, etc.) is used, and family-specific tokenization strategies are implemented to avoid bias.

Full Coverage of Measurement Positions

Measurement hooks are set at key positions such as embedding layers, hidden states, attention mechanisms, MLP/MoE modules, SwiGLU gates, and normalization layers to fully observe the propagation path of activation values.

Breadth of Model Coverage

Covers 27 checkpoints from 8 mainstream open-source model families, including Dense architectures (LLaMA, Qwen, Gemma), MoE architectures (Mixtral, Qwen-MoE), vision-language models, and versions from different training stages.

Finding 1: Cross-family differences of nearly four orders of magnitude

When the number of parameters is similar, the maximum activation values of different families differ significantly: Qwen3.5 series and MoE models are concentrated in the 10²-10³ magnitude range, while Gemma3-27B-it is as high as about 7×10⁵, challenging the intuition that "the larger the model, the larger the activation value range."

Finding 2: Natural advantages of MoE architectures

At the same scale, the maximum activation values of MoE checkpoints are 14.0-23.4 times lower than those of Dense models, possibly due to the sparse activation of the gating mechanism suppressing large values.

Finding 3: Residual streams carry the global maximum value

In 22 out of 24 checkpoints, residual streams carry the global maximum activation value. The engineering significance is that residual streams determine the boundary of the model's numerical stability.

INT-8 quantization verification shows that the measured maximum activation value and low-bit reconstruction error are significantly covariant. Choosing a scaling strategy based on actual measurements can effectively reduce information loss. Recommendations:

1. Model publishers should clearly report the maximum activation value in the model card;
2. Different model families need differentiated quantization configurations to avoid precision degradation caused by a "one-size-fits-all" approach;
3. Pre-measure the activation value distribution using representative data before deployment, instead of relying on empirical values.

Limitations

• Based on a static 5000-sample corpus, it may not capture activation behaviors of specific domains or extreme inputs;
• Focuses mainly on maximum values, without in-depth analysis of the complete distribution pattern of activation values (such as long-tail characteristics, outlier frequency).

Future Directions

• Extend the measurement framework to models with 100B+ parameters;
• Study the causal relationship between the dynamic range of activation values and training data, optimizer selection;
• Develop adaptive quantization algorithms based on activation value characteristics.

This study reveals the huge differences in the dynamic range of activation values of modern open-source large language models through systematic measurements, providing important empirical basis for low-bit quantization deployment. Core conclusion: The maximum activation value is a model attribute that should be measured and reported, not a minor detail. Model developers and deployment engineers need to include activation value analysis in standard processes. The research code has been open-sourced to help the community understand the numerical characteristics of models and balance efficiency and precision.