Study on the Diminishing Returns of Early Exit Decoding in Modern Large Language Models

This paper re-evaluates layer-wise early exit techniques in modern large language models (LLMs), finding that the effectiveness of early exit shows a diminishing trend with the evolution of model generations. Reasons include improvements in model pre-training methods and architectural innovations that reduce inter-layer redundancy, making it difficult for shallow representations to support accurate predictions. The study also proposes new metrics to quantify the early exit adaptability of models and provides practical insights and future directions.

Overview of Early Exit Techniques

Early exit is a dynamic inference acceleration technique whose core is to stop computation when a simple input forms a sufficiently confident prediction at a shallow layer. Traditional mechanisms evaluate confidence through a classification head at each layer, with advantages including reduced latency, saved computation, and adaptive adjustment of computational load.

Impact of Modern LLM Architecture Evolution

• Pre-training Improvements: Longer training steps, optimized learning rate scheduling, high-quality data filtering, etc., reduce inter-layer redundant representations. Early models had high inter-layer similarity, while modern models have substantial feature transformations at each layer.
• Architectural Innovations: RMSNorm replacing LayerNorm, SwiGLU activation function, RoPE positional encoding, GQA attention mechanism, etc., enhance feature extraction capabilities and reduce the feasibility of early exit.

Cross-Generational Comparison

Early models (e.g., GPT-2) can maintain over 90% accuracy with 30-50% reduced computation via early exit; modern models (e.g., Llama3) only achieve 10-20% computation reduction or require sacrificing more accuracy.

Impact of Model Scale

Models with over 20 billion parameters have higher early exit potential due to more redundant capacity, sufficient training, and structured representation space.

Differences Between Model Types

• Dense Transformers: Highest early exit potential
• MoE Models: Lower potential (inherently sparse activation)
• SSM Models (e.g., Mamba): Lower potential (state compression mechanism limits intermediate predictions)

Impact of Fine-Tuning

Base pre-trained models have higher early exit potential than instruction-tuned/RLHF models, as fine-tuning specializes the model and makes shallow-layer confidence calibration unreliable.

A comprehensive evaluation metric is proposed, including:

1. Early layer representation quality (shallow-layer discriminative ability)
2. Inter-layer information increment (new information added per layer)
3. Confidence calibration (matching degree between early layer confidence and accuracy)

An open-source benchmark is built based on this metric, supporting:

• Comparing early exit adaptability of different models
• Evaluating new early exit strategies
• Predicting potential benefits of specific models and workloads

Re-evaluating Strategies

• Adopt dynamic thresholds (based on input complexity)
• Combine multiple acceleration techniques such as quantization, pruning, and speculative decoding
• Customize strategies for specific tasks

Model Selection Trade-offs

• Large base models are more suitable for early exit
• MoE/SSM architectures lower the expected benefits of early exit
• Fine-tuned models need strategy adjustments or acceptance of low acceleration ratios

Future Architecture Design

• Explicitly design auxiliary tasks for early prediction
• Introduce early exit regularization during training
• Explore architectural elements compatible with early exit

Limitations

• Only focuses on text generation tasks
• Evaluated based on static datasets without considering dynamic workloads
• Insufficient analysis of the impact of hardware platform characteristics

Future Directions

• Develop new early exit mechanisms compatible with modern LLMs
• Explore learning methods to automatically discover optimal exit strategies
• Study early exit characteristics in multimodal models
• Design hardware-software co-optimized early exit solutions

This paper reveals the challenges of early exit techniques in modern LLMs, where traditional strategies become less effective as models evolve. Model optimization techniques need to adapt to new model characteristics, and the proposed evaluation metrics and benchmarks provide the community with objective evaluation tools to guide future research and practice.