SensorLLM: Enabling Large Language Models to Understand Motion Sensor Data

The SensorLLM framework proposed by Singapore's Cruise Research Group enables large language models to directly understand and analyze Human Activity Recognition (HAR) tasks by aligning motion sensor data with natural language. This成果 has been accepted by EMNLP 2025. The project is maintained by the Cruise Research Group, and the source code has been open-sourced on GitHub (link: https://github.com/cruiseresearchgroup/SensorLLM). Release date: 2026-06-08. SensorLLM addresses the problems of poor interpretability and weak cross-dataset generalization of traditional HAR models, providing a new direction for the field of multimodal learning.

Human Activity Recognition (HAR) is a core task in pervasive computing and health monitoring. Traditional methods rely on specialized deep learning models to process motion sensor data, but they lack interpretability and are difficult to generalize. Large Language Models (LLMs) have strong reasoning and transfer capabilities, but cannot understand numerical sensor data. SensorLLM proposes an innovative solution to this challenge.

The core innovation is establishing a semantic mapping between sensor data and natural language (not a simple format conversion), such as mapping a walking acceleration curve to the description "regular steps with moderate amplitude". The technical architecture includes:

1. Encoding module: Processes time series to retain key motion patterns;
2. Alignment training: Contrastive learning narrows the distance between sensor representations and text;
3. Downstream adaptation: Supports HAR tasks such as activity classification and action description generation.

SensorLLM has undergone peer review at EMNLP 2025. Experimental results:

• Leading performance on multiple standard HAR datasets;
• Better interpretability than traditional models (generates natural language explanations);
• Strong cross-dataset generalization ability, capable of handling unseen activity types.

Practical value:

• Health monitoring: Smartwatches accurately identify activities and provide interpretable recommendations;
• Elderly care: Detect abnormal events such as falls and explain the basis; Research significance: Provides an example for extending LLMs to non-text modalities, which can be generalized to fields such as environmental sensors and physiological signals.

SensorLLM breaks the modal barrier, allowing LLMs to "understand" sensor data. It has received academic recognition and has application potential. Outlook: After improving the performance of edge devices and optimizing LLM efficiency, it is expected to be widely applied in scenarios such as wearables, smart homes, and medical health monitoring.