Innovative Application Research of Multimodal Large Language Models in Video Fall Detection

This article introduces a research project on video fall detection based on Multimodal Large Language Models (MLLM), exploring the application of various prompt strategies such as zero-shot, few-shot, and chain-of-thought in fall detection and human activity recognition tasks. It aims to address the problems of traditional fall detection methods, which rely on large amounts of labeled data and have limited generalization capabilities.

Falls are a serious health threat to the elderly and one of the main causes of their injuries and deaths. Traditional fall detection methods rely on dedicated sensors or computer vision-based deep learning models, but they require large amounts of labeled data for training and have limited generalization capabilities. The emergence of Multimodal Large Language Models (MLLM) brings new possibilities to this field.

The project designs three experimental paradigms to evaluate the performance of MLLM:

1. Zero-shot Learning: Only receives task instructions and test videos, testing the model's basic visual understanding and semantic grasp. Command example: python scripts/vllm_inference.py experiment=zeroshot model=internvl model.params=8B
2. Few-shot Learning: Provides labeled example videos, supporting random selection and similarity retrieval (precomputing embeddings required: python scripts/vllm_inference.py experiment=embed, run command: python scripts/vllm_inference.py experiment=fewshot_similarity model=qwenvl model.params=8B)
3. Chain-of-Thought Reasoning: Prompts the model to generate a reasoning process. Command example: python scripts/vllm_inference.py experiment=zeroshot_cot

Video Preprocessing and Caching

• Disk Cache: Preprocessed video tensors are saved as .pt files, persistent across runs. Modifying parameters automatically creates a new cache. Command: python scripts/build_tensor_cache.py experiment=zeroshot data.cache_dir=outputs/tensor_cache
• Memory Cache: Lazy loading of the few-shot example corpus dictionary to avoid repeated reading

Model Fine-tuning

Supports LoRA fine-tuning of Qwen3-VL using the TRL library's SFTTrainer. Command: python scripts/train_sft.py training=full. Supports OmniFall and multi-source mixed datasets. Fine-tuned adapters can be loaded with: python scripts/vllm_inference.py model.params=8B lora.path=outputs/training/<run_name>/adapter lora.max_rank=8

Distributed Training

Supports DDP and DeepSpeed ZeRO-2. Command: accelerate launch --config_file config/accelerate/deepspeed_zero2.yaml --num_processes 4 scripts/train_sft.py training=full

In addition to fall detection, the model's generalization performance is evaluated by combining it with the Human Activity Recognition (HAR) task. Experimental results are saved in the following paths:

• Prediction results: output_dir/predictions/<wandb-project>/
• Evaluation metrics: output_dir/evaluation_results/<wandb-project>/

This study explores the cross-modal transfer capabilities of large language models. Key findings include the effectiveness of few-shot learning, the value of similarity retrieval, the role of chain-of-thought, and the necessity of fine-tuning. It provides a more flexible and universal technical path for fall detection systems in scenarios such as medical monitoring, smart homes, and elderly care.