JASMINE: A Pose Estimation-Based Screening System for Childhood Autism

This article introduces the JASMINE project, a privacy-first web application that uses computer vision (MediaPipe) and machine learning technologies to assist medical professionals in screening for Autism Spectrum Disorder (ASD) by analyzing children's movement patterns in videos. The core workflow of the system includes pose extraction, feature calculation, ensemble model inference, and result output, achieving 97.1% accuracy and 0.997 ROC-AUC on the MMASD dataset. Note: This project is a research demonstration, not a diagnostic tool, and results need to be confirmed by professional medical personnel.

Early screening for Autism Spectrum Disorder (ASD) is crucial for intervention, but traditional methods rely on professionals' experience and have issues such as strong subjectivity, high cost, and limited coverage. With the development of computer vision and machine learning technologies, using AI to assist in efficient and objective ASD screening has become a research hotspot, and JASMINE is an innovative practice in this direction.

The core processing workflow of JASMINE: 1. Use MediaPipe PoseLandmarker to extract 25 body key points per frame from the video; 2. Calculate 983 kinematic and statistical features; 3. Run a stacked ensemble model of Random Forest, SVM, TCN, and Transformer; 4. Return ASD probability score and interpretability analysis. In terms of technology stack, the front-end uses Next.js 16, TypeScript, etc., the back-end uses FastAPI, machine learning relies on MediaPipe Tasks, scikit-learn, etc., and real-time streaming transmission is implemented via SSE.

Feature engineering includes 233-dimensional kinematic features (joint angles, speed, distance, symmetry) and 750-dimensional statistical features (key point statistics, temporal dynamics, frequency analysis), totaling 983 features, which are dimensionality-reduced via RFECV during training. Model performance: Among single models, Random Forest performs best (accuracy 0.953), and the stacked ensemble model has comprehensive advantages, achieving 97.1% accuracy and 0.997 ROC-AUC (based on the MMASD dataset with 1374 subjects).

Core function highlights of JASMINE: 1. Real-time SSE streaming pipeline to push processing progress; 2. AI interpretability panel (feature importance, single model contribution, natural language reasoning); 3. Privacy protection (only processes 2D skeleton key points, does not store original videos or identity data); 4. Role-based portals (professional portal supports assessment/patient management, parent portal allows viewing results/managing records).

The project uses the MMASD multimodal autism dataset (1374 people, including 535 ASD children) for training and validation. Social value: Lower screening threshold (only requires ordinary videos), improve objectivity (stable and repeatable), emphasize human-machine collaboration (AI assists rather than replaces professional decisions), and enhance trust (interpretability design).

Deployment environment requires Python 3.11+, Node.js 18+, and optional Firebase project. Quick start steps: Clone the repository → Activate the virtual environment for the back-end and install dependencies → Start the FastAPI service → Install front-end dependencies and configure .env.local → Start the Next.js dev service. Demo accounts: doctor@demo.com (professional), parent@demo.com (parent), password is demo123 for both.