MIT Multimodal AI Course Project: Research on Multimodal Modeling of Tactile Perception and Grasping

The final project Tactile-Grasp of MIT 6.S985 "Modeling: Multimodal AI" course focuses on the fusion modeling of tactile perception and visual information in robotic grasping tasks, aiming to build a more robust robotic grasping model and provide new research ideas for the field of multimodal perception and physical interaction. The project is led by Cassandra Zhe, with the code repository created in February 2026 and the final version updated in early April.

MIT 6.S985 "Modeling: Multimodal AI" is a cutting-edge course that explores integrating multiple perceptual modalities such as vision, language, audio, and touch to build intelligent systems. The final project requires students to complete a full research cycle from data collection to experimental evaluation. The Tactile-Grasp project was born in this context, focusing on robotic grasping modeling with tactile and visual fusion, and the code repository reflects the evolution from course assignment to reproducible research.

Traditional robotic grasping relies on vision, but has limitations such as transparent objects, occlusion, and lighting changes, and cannot perceive physical properties like contact force. Tactile perception can directly measure contact force distribution, surface texture, etc., which complements vision. Humans integrate visual prediction and tactile feedback when grasping, and robots need similar capabilities, hence the need to fuse the two modalities.

Inferred technical route from the repository structure: The data layer includes visual and tactile multimodal datasets (collected via robotic arm platform), preprocessing includes standardization, time alignment, etc.; the baselines directory implements pure vision, pure tactile, and simple fusion baselines; the experiments directory designs evaluations for different object categories and strategies (metrics include grasping success rate, etc.); the reports directory contains project reports and documents.

Core challenges include: Modal heterogeneity (differences between high-resolution visual images and low-resolution tactile pressure distributions, etc.); time synchronization (asynchronous inputs: vision before grasping, tactile after contact); fusion strategy selection (early/mid/late fusion, attention mechanisms, etc.); simulation-to-reality transfer (domain randomization, adaptive technologies).

Academically, it provides empirical research for the interdisciplinary field of multimodal perception and physical interaction, quantitatively analyzing the role of touch in grasping stability. In applications, it can improve the operational capabilities of warehouse logistics, flexible manufacturing, and service robots; in medical scenarios, it enhances the fine operation and safety of surgical assistance and rehabilitation robots; in human-robot collaboration, it perceives unexpected contacts to ensure safety.

Reflects the characteristics of top AI education: end-to-end research training (from problem definition to result analysis); hands-on practice orientation (implementing runnable code); open source and reproducibility (hosted on GitHub, MIT license).

As a course assignment, Tactile-Grasp touches on the frontiers of robotics and AI, and multimodal perception and physical interaction are key paths for intelligent robots. We hope it provides a reference for related fields and look forward to more course projects producing high-quality open-source results.