AI Multimodal Pipeline: End-to-End Video Analysis System Based on PyTorch, ResNet-50, and YOLOv8

AI-MultiModal-Pipeline is an end-to-end multimodal machine learning pipeline project released by EricSerrano1111 on GitHub (2025). The system integrates three models: PyTorch CNN keyword recognition, ResNet-50 face detection, and YOLOv8 object tracking. Through FastAPI backend orchestration and Streamlit frontend interface, it realizes joint processing of voice, face, and object information in videos and generates structured analysis results.

Original Author and Source

• Maintainer: EricSerrano1111
• Source: GitHub (Project name: AI-MultiModal-Pipeline, Link: https://github.com/EricSerrano1111/AI-MultiModal-Pipeline)
• Release time: 2025

Core Functions

The system is designed to process voice, face, and object information in videos simultaneously. Core capabilities include:

1. Speech recognition and keyword detection (custom PyTorch CNN model)
2. Facial feature localization (ResNet-50 architecture)
3. Real-time object tracking (YOLOv8) Coordinated by FastAPI, Streamlit provides a web interface, enabling multimodal analysis without coding.

Model Composition

1. PyTorch CNN Keyword Recognition: Converts audio waveforms to Mel spectrograms (2D features), trained on a subset of Google Speech Commands, focusing on high-discrimination keywords (yes/no/stop/go).
2. ResNet-50 Face Detection: Uses ImageNet pre-trained weights for transfer learning to achieve high-precision face localization and feature extraction.
3. YOLOv8 Object Tracking: Real-time multi-object detection and cross-frame identity maintenance, balancing speed and accuracy.

Deployment Architecture

• FastAPI backend: Receives videos, coordinates model inference, assembles results, and provides API interfaces.
• Streamlit frontend: Supports video upload, displays progress, visualizes results, and allows downloading JSON reports.

Data Flow

1. Preprocessing: Decompose video into audio track (.wav) and video frames (.jpg).
2. Parallel Inference: SpeechAnalyzer processes audio, FaceAnalyzer processes frames, ObjectTracker tracks objects.
3. Result Integration: Generate a timestamp-aligned multimodal JSON report.

Development Process

1. Prototype development (Jupyter Lab): Model experiments, training validation, hyperparameter tuning.
2. Object-oriented refactoring: Modular design (VideoPreprocessor/SpeechAnalyzer, etc.), unit testing.
3. Service encapsulation: FastAPI + Streamlit deployment.

Environment Configuration

• Conda environment: Create multimodal-env via environment.yml.
• Weight management: Need to manually prepare face_resnet50.weights.h5 and custom_kws.pth; YOLOv8n.pt is downloaded automatically; paths are managed uniformly by config.yaml.

Application Scenarios

• Intelligent video surveillance: Simultaneously analyze voice, faces, and objects.
• Content moderation: Detect sensitive keywords, people, or non-compliant objects.
• Meeting recording: Extract speeches, identify participants, record objects.
• Educational videos: Analyze explanation content, expressions, and reactions.

Technical Highlights

• Audio to Mel spectrogram adaptation for CNN.
• Selecting a subset of high-discrimination datasets to balance resources and performance.
• Three-stage progressive development process.
• Independent component testing to ensure reliability.

Limitations

1. Model weights need manual management.
2. More suitable for offline batch processing; insufficient real-time stream processing capability.
3. Incomplete model version management.

Improvement Directions

1. Integrate Hugging Face Hub for automatic weight downloading.
2. Introduce message queues and asynchronous processing to improve real-time performance.
3. Improve version control and A/B testing mechanisms.