ModeSwitch-LLM: A Dynamic Mode Switching Controller for Large Model Inference on a Single GPU

ModeSwitch-LLM is a lightweight request-level inference mode switching controller. By dynamically selecting modes like FP16, quantization, and speculative decoding based on request characteristics, it achieves a 2.1x latency speedup and 51.7% energy reduction on a single A100 GPU. Its core design includes multi-mode support and low-overhead feature extraction. Moreover, the rule-based controller outperforms learning-based routers, significantly improving inference efficiency while ensuring output quality.

With the large-scale application of LLMs, inference efficiency has become a key bottleneck in resource-constrained scenarios (e.g., single-GPU deployment). Existing optimization techniques have their own applicable scenarios and trade-offs:

• FP16 half-precision: Balances precision and performance, but may lead to over-computation for simple requests;
• Quantization (INT8/GPTQ): Reduces memory usage and computation, but may lose precision;
• Speculative decoding: Accelerates generation, but depends on the quality of the draft model;
• Prefix caching: Relies on request similarity;
• Continuous batching: Requires tuning of batching strategies. This leads to the need for dynamic selection of inference modes.

ModeSwitch-LLM supports FP16, INT8/GPTQ quantization, speculative decoding, and hybrid modes (e.g., GPTQ + prefix caching). It selects modes by extracting low-overhead features such as input length, output prediction, request type, and system status. A comparison between rule-based and learning-based routing:

• Rule-based: Based on heuristic thresholds (e.g., choosing INT8 for short inputs), low overhead and high interpretability;
• Learning-based: Uses small neural networks for decision-making, but has high overhead and is prone to violating constraints. Experiments show that the rule-based controller performs better.

Experiments were conducted on an A100 GPU using Llama3.1-8B-Instruct:

• 2.1x latency speedup and 51.7% energy reduction (energy per token is 48% of FP16);
• Precision remains good, with an average difference of only +0.17 percentage points;
• Comparison with fixed modes:

  Configuration	Latency	Energy Consumption	Precision
  FP16 Baseline	1.0x	1.0x	Baseline
  Fixed INT8	1.5x	0.6x	-2.1%
  Fixed GPTQ	2.0x	0.4x	-5.3%
  ModeSwitch-LLM	2.1x	0.48x	-0.17%
  ModeSwitch-LLM balances efficiency and quality.

Design insights of ModeSwitch-LLM:

1. Request heterogeneity is key to optimization; static configurations tend to waste resources or reduce quality;
2. Simple heuristic rules are more practical than complex learning models (low overhead, high interpretability);
3. No need for model retraining or architecture modification, compatible with existing frameworks;
4. Quality gate mechanisms ensure no precision degradation, suitable for production environments.

Applicable scenarios:

• Cloud LLM services: Dynamically optimize resource allocation and reduce operational costs;
• Edge devices: Serve more users with limited resources;
• Hybrid cloud: Select optimal modes based on data sensitivity, etc. Future directions:

1. Finer-grained (token/layer-level) mode adjustment;
2. Online learning to optimize routing strategies;
3. Multi-model collaborative routing;
4. Co-design with AI accelerator hardware.

ModeSwitch-LLM achieves a balance between inference efficiency and quality on a single GPU through lightweight dynamic mode switching. Its work emphasizes the importance of system design and heuristic optimization in engineering practice. As LLM applications become more widespread, such dynamic optimization technologies will play a key role in AI infrastructure, and we look forward to more practical system validations and expansions.