SEATS: Phased Adaptive Token Selection Optimization for Multimodal Large Language Models

Multimodal large language models (om-LLMs) can understand video, audio, and text simultaneously, but they incur huge computational overhead when processing dense non-text tokens. Researchers propose the SEATS phased adaptive token selection method, which achieves training-agnostic efficient inference based on the block-level decay pattern of inter-layer token dependencies. When retaining only 10% of audio-visual tokens, SEATS reduces FLOPs by 9.3x while maintaining 96.3% performance, providing a key optimization for the practical deployment of om-LLMs.

om-LLMs encode video and audio into time-series tokens and interleave them with text inputs to achieve deep fusion, but this brings computational challenges: the number of tokens for a 10-second video (30fps) plus audio and text can reach thousands, making the computation and memory usage unbearable when processed through dozens of Transformer layers. Existing token pruning methods have limitations: visual centrism (ignoring the uniqueness of audio), static strategies (unable to adapt to inter-layer dynamics), and neglect of the evolutionary laws of cross-modal fusion.

Researchers found that the inter-layer dependencies of visual and audio tokens in om-LLMs exhibit a block-level pattern and decay with depth: more tokens need to be retained in shallow layers to support cross-modal alignment; in middle layers, modal information is gradually fused, so redundant tokens can be removed; after cross-modal fusion is completed in deep layers, most non-text tokens can be discarded. This provides a theoretical basis for dynamically adjusting the token retention strategy (conservative in shallow layers, progressive in middle layers, and aggressive in deep layers).

SEATS is a training-agnostic phased framework: 1. Pre-input spatio-temporal redundancy elimination: Remove visual (spatio-temporal redundancy) and audio (time-frequency redundancy) tokens via attention-weighted diversity selection; 2. Progressive pruning inside LLM: Dynamically allocate budgets (based on query relevance scores) after each Transformer block to gradually remove unimportant tokens; 3. Full pruning in deep layers: Remove all non-text tokens in the deep layers of LLM (last 1/4 to 1/3 layers), as fused information has been encoded into text representations.

Evaluated on Qwen2.5-Omni and Qwen3-Omni: The aggressive configuration (retaining 10% of audio-visual tokens) achieves a 9.3x reduction in FLOPs, a 4.8x acceleration in Prefill, and maintains 96.3% performance; Trade-offs for different retention rates: 50% retention →98.5% performance /3.2x FLOPs reduction, 25%→97.4%/5.8x,5%→92.1%/12.6x. Performance loss is mainly in fine-grained perception tasks, while higher-level semantic tasks have a higher retention rate.

Implications of SEATS:1. Break the traditional trade-off between computation and quality, and find better points on the Pareto frontier;2. The training-agnostic feature lowers the integration threshold, enabling plug-and-play;3. Reveal the layered functions of cross-modal fusion (shallow alignment, middle-layer fusion, deep-layer reasoning), providing guidance for model architecture design.

Limitations of SEATS: Task sensitivity (more sensitive to fine-grained perception tasks), complex long video processing, and simple audio processing. Future directions: Task-aware pruning (dynamically adjust strategies), hierarchical time modeling (multi-scale time representation), and joint optimization (combining token selection with architecture search).