DAT: Efficient Inference and Adaptive Transmission Scheme for Multimodal Large Models in Edge Cloud Systems

Addressing the high overhead issue of continuous video stream processing in bandwidth-constrained edge cloud environments, DAT proposes a collaborative small-large model cascading architecture and a semantic-and-bandwidth-aware multi-stream adaptive transmission optimization method, achieving 98.83% recognition accuracy and a 77.5% reduction in alert latency, thus providing a solution for the efficient deployment of multimodal large models in edge clouds.

Background and Challenges

Multimodal Large Language Models (MLLMs) excel in semantic understanding and visual reasoning capabilities, but when processing continuous video streams in bandwidth-constrained edge cloud environments, the computational and communication overhead is extremely high, hindering low-latency alerts and effective visual evidence transmission. Traditional solutions either perform full inference at the edge or transmit all data to the cloud, both of which have efficiency bottlenecks.

Overall Architecture Design of DAT

Researchers propose the DAT (Dual-Aware Adaptive Transmission) framework, which balances high-quality semantic generation, low-latency alerts, and visual evidence supplementation through intelligent pre-screening and elastic transmission strategies. Its core lies in endowing the edge with lightweight intelligent decision-making capabilities to filter invalid data and only submit content requiring in-depth analysis to the cloud's large model.

Collaborative Small-Large Model Cascading Mechanism

First major innovation of DAT: The lightweight small model at the edge acts as a gating module, filtering non-target frames and detecting suspicious events, only triggering cloud MLLM inference to reduce unnecessary deep inference costs.

Visual Guidance and Semantic Prompt Fine-Tuning Strategy

By combining visual guidance (focusing on key regions) and semantic prompts (understanding high-level semantics) to fine-tune the small model, it improves structured event understanding, target detection, and output consistency, reducing false and missed reports.

Semantic and Bandwidth-Aware Multi-Stream Adaptive Transmission

Second major innovation of DAT: Dynamically adjust data stream priority and encoding strategies based on network congestion and semantic urgency. Prioritize the transmission of time-sensitive alert information, and use efficient compression for visual evidence transmission to ensure service quality under different network conditions.

Experimental Evaluation and Performance

Multi-scenario evaluations show: DAT achieves 98.83% recognition accuracy and 100% output consistency; under severe congestion, the weighted semantic alert latency is reduced by 77.5%, and 98.33% of visual evidence is delivered within 0.5 seconds, verifying the effectiveness of the joint optimization of cascaded inference and elastic transmission.

Technical Insights and Future Outlook

DAT demonstrates the possibility of improving efficiency through system-level optimization (not just model compression), providing a reference for the deployment of multimodal large models in edge clouds. In the future, with the enhancement of edge computing capabilities and the development of network technologies, collaborative optimization solutions are expected to be applied in more scenarios, promoting the efficient and reliable evolution of intelligent visual systems.