Multimodal Deep Learning for Metabolic Consumption Prediction: Intelligent Fusion of Wearable Devices and Biometric Features

This article introduces the open-source project close_loop_diet_recom, which uses a multimodal deep learning framework to fuse real-time signals from wearable devices (such as PPG and accelerometer data) with users' static biometric features (like age and body fat percentage) to accurately predict oxygen consumption (VO2). It provides data support for scenarios like personalized health management, dietary recommendations, and exercise guidance, aiming to address the high cost of traditional VO2 measurement and the insufficient accuracy of VO2 estimation by consumer-grade wearables.

Oxygen consumption (VO2) is the gold standard for measuring energy metabolism, and it is crucial for optimizing athlete training, managing the diet of chronic disease patients, and formulating calorie plans for weight loss individuals. Traditional VO2 measurement relies on lab equipment (such as a mask + treadmill), which is costly and has a poor user experience; while consumer-grade wearables can provide basic data, their direct VO2 estimation accuracy is insufficient. The close_loop_diet_recom project aims to fill this technical gap.

The project's core is a multimodal data fusion architecture:

1. Dynamic Sensor Signal Stream: Processes real-time signals from wearables (PPG, accelerometer, temperature, etc.) and extracts dynamic features via a temporal encoder;
2. Static Biometric Layer: Integrates static information like age, gender, and body fat percentage, encoding personalized prior knowledge through an embedding layer;
3. Fusion Mechanism: Uses an attention mechanism (cross-attention) to enable interaction between dynamic and static features, automatically learning feature contribution weights; The model architecture includes a lightweight temporal encoding module (1D convolution + GRU), a cross-attention fusion layer, and an MLP prediction head with residual connections, balancing accuracy and computational efficiency.

The project designs training strategies to address real-world data issues:

• Data Alignment: Uses a sliding window strategy to resolve sampling frequency differences and timestamp drift across multi-source sensors;
• Missing Value Handling: Introduces a mask training strategy to improve the model's robustness when data is missing;
• Domain Adaptation: Enhances the model's generalization ability across wearable device brands via domain adversarial training.

The application scenarios of this technical framework include:

1. Closed-Loop Dietary Recommendations: Dynamically adjusts calorie and nutrition suggestions based on real-time VO2;
2. Exercise Guidance: Precisely determines whether exercise intensity is within the target range (fat burning/aerobic);
3. Chronic Disease Management: Assists in insulin adjustment for diabetic patients and condition monitoring for heart failure patients;
4. Clinical Research: Provides a low-cost, highly accessible metabolic monitoring solution to support large-scale population studies.

The project has the following limitations and improvement directions:

• Calibration Requirement: Requires a small amount of user data for fine-tuning to achieve optimal accuracy; needs to explore personalized methods under privacy protection;
• Robustness in Extreme Scenarios: Signal quality degrades during high-intensity exercise or abnormal physiological states; needs to improve prediction stability;
• Energy Consumption Optimization: The battery life issue of running complex models on wearables requires model compression and edge optimization.

As an open-source project, close_loop_diet_recom provides model code, data preprocessing pipelines, and evaluation benchmarks, lowering the barrier for subsequent research in the health AI field. It is an excellent starting point for developers and researchers to enter the wearable health AI domain.

Multimodal deep learning is rapidly evolving in the health monitoring field. The close_loop_diet_recom project breaks through the bottleneck of single-modal approaches by fusing complementary data sources. With the popularization of wearable devices and advances in sensor technology, such personalized health AI tools are expected to move from labs to the general public, realizing the vision of precise health management.