Multimodal Vitamin Deficiency Prediction System: Deep Learning Practice Integrating Visual and Sequential Data

This project aims to address the problems of high cost and strong invasiveness in traditional vitamin deficiency diagnosis. It builds an end-to-end multimodal deep learning pipeline that integrates CNN image analysis (e.g., photos of tongue coating, nails, etc.) and LSTM/GRU sequential modeling (lifestyle data), and realizes intelligent prediction through a Streamlit interactive interface, providing a low-cost, non-invasive solution for early screening in the health field.

Vitamin deficiency is a global health issue, but traditional diagnosis relies on blood tests, which have the drawbacks of high cost and strong invasiveness. A single data source cannot fully reflect nutritional status: image data (tongue coating, nails, etc.) can capture visible symptoms but are greatly affected by individual differences and shooting conditions; lifestyle data (diet, work-rest schedule, etc.) reflect long-term patterns but are sequential and require modeling of time dependencies. Multimodal fusion allows the two to complement each other, forming a more reliable basis for prediction.

The project adopts an encoding-fusion-decoding architecture:

• Visual Encoding Branch: Uses CNN to process images, possibly based on pre-trained ImageNet backbone for transfer learning and fine-tuning, automatically learning hierarchical visual features (edge texture → shape pattern → symptom semantics) to identify subtle deficiency signs.
• Sequential Encoding Branch: Uses LSTM/GRU to process lifestyle sequential data, alleviates gradient vanishing through gating mechanisms, and captures long-range sequential patterns; GRU, as a lightweight variant, is more suitable for rapid deployment.
• Fusion Strategy: Fuses at the feature level, with options of early, late, or intermediate fusion. The core is to mutually enhance visual and sequential information at an appropriate abstract level (specific strategies depend on implementation details).

Engineering considerations for the end-to-end pipeline:

• Data Preprocessing: Standardize image size, normalize pixels, possibly perform data augmentation; handle missing values, align windows, and conduct feature engineering for sequential data.
• Model Training: Balance multiple loss functions, use modal dropout to enhance robustness.
• Inference Service: Consider latency and concurrency, possibly quantize/distill the model to adapt to edge devices.

Value of the Streamlit interactive interface: Lowers the threshold for use (non-technical users can upload photos and fill out questionnaires to get results); displays interpretability (heatmaps, factor contribution degrees); supports rapid iteration (adjust the interface with declarative syntax).

Challenges faced by the project:

• Data Quality: The accuracy of training data annotations and the representativeness of distribution affect reliability; it is necessary to verify the correspondence between image/questionnaire data and the gold standard of blood tests.
• Privacy Protection: Health data is sensitive and requires strict encryption and access control.
• Regulatory Compliance: Medical AI requires clinical trials and approval; there is a gap between the prototype and the formal product.
• Responsibility Boundary: The output should be clearly stated as "auxiliary screening" to avoid user misunderstanding and delay in formal medical treatment.

This project demonstrates the potential of AI in preventive medicine, providing low-cost non-invasive assessment by fusing easily accessible data sources. Its CNN+RNN+multimodal fusion architecture is a successful application scenario of multimodal learning. For medical AI or multimodal developers, it provides a full-process reference implementation from data processing to deployment.