Multimodal Brain Visual Cortex Model: Exploring the Intersection of Neuroscience and AI

The multimodal-brain-scaling project of EPFL NeuroAI Lab focuses on the intersection of neuroscience and AI, building more accurate computational models of the visual cortex through multimodal data and task optimization. The core of the research revolves around the neural mechanisms of multimodal integration, the scaling laws of visual models, and the impact of task optimization, aiming to bridge the visual processing mechanisms of the brain and AI model design, and promote the development of both fields bidirectionally.

Understanding the brain's visual information processing is a core issue in neuroscience. Over the years, the structure of the visual cortex has been revealed through experiments and modeling. Meanwhile, deep learning models have made breakthroughs in image recognition. EPFL NeuroAI Lab is committed to using AI to understand the brain and gain inspiration for AI design from brain mechanisms. The multimodal-brain-scaling project is the result of this effort, exploring the construction of visual cortex models through multimodal data and task optimization.

1. Neural Mechanisms of Multimodal Integration

The visual cortex integrates information such as motion, depth, and color. This section explores the representation methods of different modalities, integration principles, and simulation models.

2. Scaling Laws

Explore the changing patterns of visual model performance with scale, data volume, and computational volume, whether they apply to neural data prediction, and optimal configurations.

3. Impact of Task Optimization

Analyze the impact of different visual tasks (object recognition, scene understanding, etc.) on neural representations, and compare the effects of multi-task learning and self-supervised learning.

Model Architecture

• Visual Transformer (ViT): Uses self-attention to process image patches, exploring the impact of patch size, number of layers, and positional encoding.
• CNN: ResNet series, with different depth and width variants corresponding to biological visual hierarchies.
• Multimodal Fusion: Early, middle, and late fusion methods.

Datasets and Evaluation

• Neurophysiological datasets: V1/V2 electrophysiological recordings, fMRI, MEG/EEG data.
• Evaluation metrics: Neural prediction accuracy, Representational Similarity Analysis (RSA), hierarchical correspondence.

Training Strategies

• Multi-task learning: Simultaneously optimize tasks such as image classification and object detection.
• Self-supervised learning: Contrastive learning, masked image modeling, multimodal contrastive learning.

1. Advantages of Multimodal Training

Multimodal models outperform unimodal ones in predicting neural responses—for example, motion information improves MT area prediction, and depth information enhances the dorsal pathway.

2. Optimal Model Scale

There exists an optimal "sweet spot" for model scale; different brain regions have different requirements, and computational efficiency and accuracy need to be balanced.

3. Importance of Task Selection

Scene understanding tasks produce comprehensive representations, fine-grained classification optimizes object recognition areas, and multi-task combinations are more effective.

4. Hierarchical Correspondence

The shallow layers of the model correspond to V1, middle layers to V2/V4, and deep layers to the IT area; multimodal models have more stable correspondences.

Neuroscience Research

Provide models to validate hypotheses, generate experimental predictions, and integrate cross-modal neural data.

AI Model Design

Gain architectural inspiration from the brain, develop efficient multimodal algorithms, and improve generalization and robustness.

Clinical Applications

Understand the mechanisms of visual disorders, develop neural prosthetic models, and assist in brain-computer interface design.

Current Limitations

• Models are based on static images; research on dynamic processing is limited.
• Neural data comes from primates; cross-species generalization needs verification.
• Computational resource constraints limit large-scale experiments.

Future Directions

• Integrate more modalities such as touch and hearing.
• Explore temporal dynamics and attention mechanisms.
• Develop lightweight models for real-time applications.
• Establish standardized evaluation benchmarks.

This project represents the cutting edge of the intersection between neuroscience and AI. By studying visual cortex modeling through multimodal data and task optimization, it provides a new perspective for understanding the brain's visual mechanisms and points the way for AI visual system design. This paradigm of using large-scale computational models and diverse data to study the brain is becoming a new standard in neuroscience; in the future, it will simulate brain functions more accurately and achieve bidirectional promotion.