HOI-MLLM: Open-World Human-Object Interaction Detection Based on Multimodal Large Language Models

The HOI-MLLM project combines multimodal large language models (MLLMs) with chain-of-thought (CoT) reasoning to achieve open-world human-object interaction (HOI) detection, breaking through the limitations of traditional methods in understanding complex scenarios. Developed and open-sourced by jasminethurder, this project represents an important attempt to advance HOI research toward generality and flexibility.

Human-Object Interaction (HOI) detection is a core problem in computer vision. Traditional methods rely on predefined interaction categories and annotated data, performing well on closed datasets but struggling in open-world scenarios. Real-world interactions are complex and diverse, requiring an understanding of semantic relationships. The rise of multimodal large language models—with their ability to process visual and textual information simultaneously and describe complex interactions—provides new possibilities for open-world HOI detection.

HOI-MLLM is an open-source project that combines MLLMs with chain-of-thought reasoning to solve open-world HOI detection. Its core technologies include: 1. Multimodal fusion mechanism: Advanced encoders map image and text features into a unified semantic space, enabling deep interaction via attention mechanisms; 2. Chain-of-thought reasoning: Guides the model to reason step-by-step (identify humans and objects → analyze spatial relationships → infer interaction types), improving accuracy and interpretability; 3. Open-world extension: Handles unseen interaction types, generates natural language descriptions, and uses external knowledge to understand complex semantics.

The advantages of HOI-MLLM include: 1. Breaking closed-category limitations: Leveraging MLLMs' semantic capabilities to describe almost unlimited interaction behaviors; 2. Interpretable reasoning process: Chain-of-thought reasoning shows intermediate steps, enhancing decision transparency; 3. Zero-shot and few-shot capabilities: Relies on MLLMs' pre-trained knowledge to quickly adapt to new tasks even with limited annotated data.

Application scenarios of HOI-MLLM include: 1. Intelligent monitoring and security: Detecting abnormal interactions (e.g., lock picking, assisting the elderly); 2. Robot vision and interaction: Helping robots understand human operations and provide assistance; 3. Autonomous and assisted driving: Identifying interactions between pedestrians and the environment (e.g., crossing the road, avoiding obstacles); 4. Video content understanding and retrieval: Fine-grained semantic annotation to support natural language queries.

Current limitations: High inference latency makes it hard to meet real-time requirements; poor performance in dense crowds or occluded scenes; insufficient understanding of professional domain interactions due to reliance on basic MLLM capabilities. Future directions: Develop lightweight models to improve speed; introduce temporal information to support video interaction detection; integrate knowledge graphs to enhance professional domain understanding; explore new multimodal fusion mechanisms (e.g., depth information, event camera data).

HOI-MLLM represents an important step toward the development of HOI detection in the direction of multimodality, open-world capability, and interpretability. By combining MLLMs' understanding capabilities with chain-of-thought reasoning, it breaks through traditional limitations and provides a new paradigm for the fusion of computer vision and language. With the advancement of large models and improvements in computational efficiency, its subsequent work will bring greater breakthroughs in human-object interaction understanding, supporting fields such as intelligent robots and autonomous driving.