TokenTriage: Eliminating the "Overthinking Tax" in Large Model Inference via Adaptive Token Budget Allocation

TokenTriage effectively addresses the "overthinking tax" problem caused by treating all queries equally in large language model inference through a lightweight query difficulty classifier and dynamic token budget allocation mechanism. It significantly reduces inference costs while maintaining output quality. This solution is applicable to multiple scenarios such as enterprise customer service, code assistance, and educational tutoring, providing an efficient optimization path for large-scale LLM deployment.

Current mainstream LLMs (e.g., GPT-4, Claude, Llama) use a fixed computation mode during inference, generating roughly the same number of tokens regardless of whether the query is simple or complex. This "one-size-fits-all" strategy leads to redundant token consumption for simple questions (such as business hours queries in customer service scenarios), forming the "overthinking tax". Studies show that 60-70% of queries in practical applications are simple/medium difficulty and can be answered satisfactorily with fewer tokens, but traditional strategies cannot distinguish complexity, resulting in resource waste.

The core innovation of TokenTriage lies in its lightweight query difficulty classifier and hierarchical budget strategy:

1. Lightweight Feature Extraction: Quickly assesses query complexity from four dimensions—vocabulary complexity (density of technical terms), syntactic structure (sentence length/nesting depth), semantic features (question type), and context dependency (whether multi-step reasoning is required)—taking milliseconds.
2. Dynamic Budget Allocation: Allocates token budgets based on classification results: minimal budget for simple queries (concise answers), medium budget for medium queries (moderate explanations), and sufficient budget for complex queries (multi-step reasoning), achieving resource-demand matching.

TokenTriage's technical implementation includes three main components:

1. Query Classifier: Uses a lightweight GBDT model, which has the advantages of fast inference speed (tens of microseconds), strong interpretability, and low resource consumption. It is trained using labeled query-difficulty pairs.
2. Token Budget Control: Achieves precise control through prompt instructions (e.g., "Answer concisely in one sentence"), adjusting generation parameters (temperature/Top-p), and dynamic max_tokens limits.
3. Feedback Loop: Monitors token deviations and user feedback, and retrains the classifier regularly to improve accuracy.

TokenTriage has verified its effectiveness across multiple scenarios:

• Enterprise Customer Service: Token consumption for simple questions is reduced by 50-70%, while complex questions still receive sufficient answers;
• Code Assistance Tools: Balances answer quality and operational costs;
• Educational Tutoring: Adjusts the depth of explanation based on question difficulty, avoiding information overload or insufficient explanation.

TokenTriage complements existing optimization technologies:

• vs Model Quantization/Distillation: Maintains the integrity of the base model without precision loss and can be used in combination;
• vs Speculative Decoding: Focuses on reducing the number of tokens (rather than accelerating generation), with complementary optimization dimensions;
• vs Caching Mechanism: Handles new questions and complements caching (for repeated questions).

Limitations: The accuracy of the classifier directly affects the effect (misjudgment leads to resource waste or quality degradation); some query difficulties are hard to pre-judge (e.g., simple questions with hidden complex boundaries). Future Directions: Expand to multi-modal inference scenarios, personalized budget allocation, and customize classification features for different model architectures.

TokenTriage provides an elegant and practical solution to the "overthinking tax" in LLM inference, helping enterprises reduce operational costs and improve user experience. As LLM applications become more widespread, adaptive inference will become an important optimization direction, worthy of continuous attention and exploration.