MotionCore: An Intelligent Dance Movement Analysis and Teaching System Based on Large Language Models

MotionCore is an open-source dance movement analysis system integrating computer vision and large language models (LLMs). Its core functions include extracting 3D skeleton sequences via MediaPipe, generating real-time streaming analysis reports, and audio-aligned dual-video synchronous comparison playback, providing an intelligent solution for dance teaching and movement correction. Its design concept is "comparative learning": users upload their own movement video and a standard video, and the system automatically analyzes differences and gives improvement suggestions, representing a new direction for AI-assisted physical education teaching.

Traditional dance teaching software only provides visual posture comparison and lacks intelligent analysis capabilities. Addressing this pain point, MotionCore adopts a dual-modal fusion design of "visual perception + language understanding", incorporating the cognitive capabilities of LLMs to understand movement details, identify problems, and provide natural language guidance like a professional coach. The system is positioned as an open-source tool to serve dance learners, coaches, and enthusiasts, lowering the learning threshold.

MotionCore uses a layered architecture:

• Frontend Interaction Layer: Built with HTML5/CSS3/JS, including video upload area, real-time preview area, streaming report area, synchronous player, and supports Chinese-English switching;
• Backend Processing Layer: FastAPI framework provides asynchronous API services, including endpoints for upload, real-time streaming, progress query, etc.;
• Core Algorithm Module: MediaPipe Pose extracts 33 3D key points, YOLO object detection for preprocessing, MoviePy + NumPy for audio alignment, and integration of LLMs such as OpenAI/DeepSeek/Gemma.

The system's typical flow is a closed loop of "upload-process-analyze-compare":

1. Video Upload: Users upload their own movement video (Video A) and a standard video (Video B);
2. Skeleton Extraction: MediaPipe extracts key points frame by frame, which users can view in real time via MJPEG stream;
3. Streaming Analysis: LLMs generate reports containing movement completion rate, joint angle comparison, rhythm matching degree, and improvement suggestions (output via SSE streaming);
4. Synchronous Playback: Dual videos play synchronously after audio alignment. It also supports multi-language interfaces and reports, and allows switching between LLM providers.

Three innovative points of the system:

1. LLM Understanding of Time-Series Data: Structured encoding of 3D skeleton sequences into text, enabling LLMs to "understand" movements;
2. Streaming Generation Experience: SSE technology实现逐字输出 of reports, enhancing user immersion;
3. Audio-Driven Alignment: Matching audio offsets based on music beats to ensure rhythm synchronization in comparison playback.

MotionCore has a wide range of application scenarios:

• Dance Teaching: AI teaching assistant enables one-to-many personalized guidance;
• Fitness Training: Evaluation of movement standards for yoga, Pilates, etc.;
• Sports Training: Posture correction for martial arts, gymnastics;
• Rehabilitation Medicine: Evaluation of movement standardization in physical therapy;
• Movement Research: Data collection tool for dance studies and human kinematics.

Limitations: Self-occlusion in complex movements affects detection accuracy; MediaPipe's 3D depth accuracy is limited; high-resolution videos require strong GPU support; Future Directions: Multi-view fusion to improve 3D reconstruction accuracy; explore end-to-end understanding of video large models; develop mobile versions; build dance movement datasets to support style transfer.