FoodSense: A Multimodal Dataset and Benchmark Model for Predicting Multisensory Experiences from Food Images

This article introduces the FoodSense dataset (containing 66,842 human-annotated entries covering four sensory dimensions: taste, smell, texture, and sound), aiming to fill the gap in AI food understanding where deep cognitive awareness of sensory experiences is lacking; it trains the FoodSense-VL vision-language model to enable multisensory reasoning and discusses its application scenarios and cognitive science significance.

Humans can evoke multisensory experiences through food images (cross-sensory perception in cognitive science), but current AI food research is limited to recognition tasks (dish category, ingredient composition, nutrition estimation), lacking deep cognition of food sensory experiences, leading to superficial understanding.

The FoodSense dataset contains 66,842 participant-image pairs, covering 2,987 unique food images; the annotation design includes numerical scores (1-5 Likert scale to quantify sensory intensity) plus free-text descriptions (to capture subtle experiences), covering four sensory dimensions; the data covers different cultures and cooking styles to ensure the model's generalization ability.

Using large language models to expand short annotations into image-anchored inference trajectories that explain the basis of sensory predictions (e.g., inferring caramelized aroma from caramelized color); these trajectories link to visual content, providing rich training signals for the model and aiding explainability.

An end-to-end vision-language architecture is adopted, with training objectives including score prediction (regression task, mapping visual features to sensory intensity) and explanation generation (conditional text generation, integrating visual information and sensory knowledge); the two tasks collaborate: explanation generation improves score accuracy, and score prediction constrains the concreteness of explanations.

Traditional captioning metrics (BLEU, CIDEr) ignore the accuracy of sensory descriptions and consistency with images; it is suggested that future evaluations focus on the consistency between descriptions and images, the accuracy of sensory attributes, and the rationality of the inference process.

Application scenarios include intelligent catering recommendations (based on sensory preferences), virtual taste testing (enhancing immersion), food marketing (generating appealing descriptions), and dietary health management (combining nutrition and sensory preferences); this research bridges cognitive science and AI, and future AI may approach human-level food understanding.