Real-time Anomaly Detection System for UAVs Based on Large Language Models: Architecture Analysis of a Full-Stack Video Analytics Platform

Toronto Metropolitan University's NG06 Engineering Graduation Project developed a full-stack real-time video analytics platform that combines Vue.js frontend, Python backend, and LLM inference engine to achieve real-time video stream anomaly detection and visual monitoring for UAVs. The system, designed with a layered architecture, has good scalability and provides intelligent solutions for security patrols, disaster rescue, and other fields.

With the rapid development of UAV technology, the demand for real-time video monitoring and anomaly detection is growing in fields such as security patrols, disaster rescue, and industrial monitoring. Traditional video monitoring relies on manual inspection, which is inefficient and prone to missing key information. This project introduces large language models into the video frame processing workflow to achieve intelligent anomaly detection and real-time feedback, addressing the pain points of traditional monitoring.

The core architecture of the system is divided into three modules: frontend display layer, backend inference layer, and streaming media transmission layer. The frontend uses Vue.js + TypeScript to build the monitoring dashboard; the backend uses Python 3.12+ to handle video stream reception and anomaly detection; video streams are pushed via OBS's WHIP protocol (an emerging WebRTC streaming standard), which has lower latency and better browser compatibility. The frontend and backend communicate via RESTful API and WebSocket to ensure real-time data push. The technology selection focuses on environment consistency (Python virtual environment) and detailed documentation, recording dependency versions and installation steps such as Node.js 24.13.0+ and Python 3.12+.

The biggest highlight of the project is the integration of LLM into the video analysis workflow: traditional object detection only identifies predefined object categories, but with the introduction of LLM, deep semantic understanding and reasoning of scenes can be achieved. In implementation, after video frames are processed by the object detection model, the results are sent to LLM for further analysis. LLM can understand the meaning of abnormal behaviors based on context and provide more intelligent detection insights. This "detection + understanding" two-layer architecture significantly improves the accuracy and interpretability of anomaly detection.

Deployment process: 1. Clone the repository; 2. Configure the backend Python virtual environment and frontend Node.js environment; 3. Configure OBS's WHIP service to point to the local server's /whip endpoint, and set audio and video encoding parameters (H.264 video, Opus audio, 5000 Kbps bitrate); 4. Push local video or real-time camera footage to the backend for processing.

Application scenarios: Real-time analysis of video streams during UAV patrols, automatic marking of abnormal objects/behaviors and pushing to the monitoring dashboard via WebSocket, reducing manual burden. Expansion potential: The modular design facilitates the integration of more AI models later or connection to actual UAV image transmission systems; the backend API is clear (endpoints for detection data, statistical data, anomaly queries, etc.), and the frontend component-based development is conducive to adding new visualization functions.

This project demonstrates a highly completed student engineering practice case, which reflects professionalism from requirement analysis to document writing. The innovative idea of introducing LLM into video analysis provides a reference for similar projects. For developers, this project is a high-quality teaching material for learning full-stack development, allowing in-depth understanding of best practices for Vue frontend, Python backend, WebRTC streaming media, and AI model integration.