MotionCore: An Intelligent Dance Analysis System Based on Pose Estimation and Large Language Models

MotionCore is an open-source intelligent dance analysis tool that integrates computer vision (pose estimation) and large language models. It supports dual video uploads, 3D skeleton extraction, real-time AI report generation, and audio-video synchronization comparison. It addresses pain points in traditional dance learning—such as low efficiency of subjective comparison and difficulty in accurately locating movement differences—providing an intelligent solution for dance teaching and learning.

Dance learning relies on visual comparison and movement detail analysis. In traditional methods, learners need to repeatedly watch instructional videos and subjectively compare their practice videos, which is inefficient and makes precise difference localization hard. With the development of computer vision and AI technologies, combining pose estimation, 3D skeleton reconstruction, and large language models has become a new direction for dance teaching. MotionCore is an open-source project developed based on this idea.

MotionCore adopts a front-end and back-end separation architecture: the front-end is built with native HTML5, CSS3, and JavaScript without complex tools; the back-end provides asynchronous API services based on FastAPI. For computer vision, MediaPipe Pose is used to extract 33 3D skeleton key points; the audio alignment module combines MoviePy and NumPy to achieve beat synchronization of dual videos; it integrates large language models from multiple vendors (OpenAI GPT series, DeepSeek, locally deployed Gemma4), allowing flexible switching by users.

1. Dual Video Upload and Skeleton Extraction: Supports drag-and-drop/click upload of practice and instructional videos. The backend automatically calls MediaPipe to extract 3D skeleton sequences per frame, displaying processing progress and remaining time in real time.
2. Streaming AI Analysis Report: Inputs skeleton data into large language models and outputs reports via Server-Sent Events (SSE) streaming, covering movement accuracy, rhythm matching, key frame suggestions, etc., and can be interrupted at any time.
3. Audio-Video Synchronization Comparison Player: Achieves dual video synchronization via audio alignment algorithms, enabling side-by-side viewing to intuitively compare movement differences.
4. Multi-Language and Multi-Model Support: Interface switches between Chinese and English, with AI report language adjusting synchronously; compatible with multiple models—DeepSeek is suitable for Chinese users, and Gemma4 can be locally deployed to protect privacy.

• Dance training institutions: Assist teachers in quickly generating movement analysis reports for students, improving teaching efficiency.
• Self-learners: Obtain professional-level movement comparison and feedback,弥补ing the lack of guidance in self-learning.
• Dance researchers: Output skeleton sequence data can be used for academic research such as movement pattern analysis and style recognition.

Deployment process: Clone the code repository → Create a Python 3.10 virtual environment → Install dependencies → Copy the environment variable template and fill in the API key → Start the main program and access the local port 8000. The project provides detailed API documentation, including descriptions of endpoints for video upload, progress query, streaming analysis, etc., to facilitate secondary integration by developers.

MotionCore is an innovative application of AI technology in art education, connecting technical tools with humanities and art. In the future, the development of multimodal models is expected to support more complex movement analysis, natural interaction, and real-time video call movement guidance; as an open-source project, it provides a complete technical reference for cutting-edge AI technologies to land in vertical application scenarios.