Farewell to Uniform Token Processing: A New Paradigm of Adaptive Compression for Time-Series Language Models

Researchers found that time-series tokens and prompt tokens have fundamentally different information structures, and proposed an adaptive token budget framework. By compressing time-series tokens via frequency-domain structure and reducing prompt tokens layer by layer, they achieved an inference speedup of up to 7.68x, providing a new direction for the efficient design of time-series language models.

When large language models expand into the time-series domain, the mainstream uniform token processing method ignores the information structure differences between time-series tokens and prompt tokens. Key findings include: the spectral contribution of time-series tokens is highly uneven, with a lot of redundancy; the influence of prompt tokens gradually decays as the model depth increases, so it is unnecessary to retain complete prompt tokens in deep layers.

The framework optimizes token usage from two aspects: 1. Compress time-series tokens based on frequency-domain structure, identify redundant parts and safely compress/discard them while retaining key temporal evidence; 2. Reduce prompt tokens layer by layer—keep complete prompt information in shallow layers and gradually reduce them in deep layers to free up computing resources.

Validated on time-series tasks such as prediction, classification, imputation, and anomaly detection: achieved an inference speedup of up to 7.68x, improved performance in 78% of evaluation settings, and performed excellently across multiple task types.

The framework is essentially a redistribution of information entropy, concentrating computing resources on valuable tokens; it also aligns with the selective attention mechanism that humans use to process time series, simulating how humans focus on key features and ignore redundancy.

The 7.68x speedup supports real-time time-series analysis (e.g., high-frequency trading, industrial monitoring); reducing the number of tokens lowers resource requirements, facilitating deployment on edge devices; it provides an efficient path for the fusion of time series and text, promoting the development of multimodal applications in finance, healthcare, etc.

Current limitations: frequency-domain analysis has insufficient stability for non-stationary/irregular time series; adaptive budget requires task-specific tuning; the interpretability of compression decisions needs to be improved. Future directions: dynamic budget allocation, cross-modal compression expansion, end-to-end learning of optimal strategies.

This study challenges the traditional paradigm of uniform token processing, reveals the information structure differences between time-series and prompt tokens, achieves significant speedup through the adaptive framework, provides new ideas for the efficient design of multimodal foundation models, and points the way to building faster and more efficient AI systems.