Multimodal AI Empowers Early Screening for Dyslexia: Innovative Fusion of Handwriting and Eye-Tracking Data

Dyslexia affects approximately 10% of the global population, making early identification and intervention crucial. Traditional screening relies on professional observation and standardized tests, which have issues like high costs and limited coverage. A recent open-source project demonstrates an AI framework that combines handwriting images, eye-tracking signals, and a multimodal fusion model to achieve more efficient and objective risk detection.

The core characteristic of dyslexia is reading ability significantly lower than peers (with normal intelligence and appropriate education). It is usually detected during school age, but early signs are easily overlooked. Traditional methods include standardized reading tests, cognitive assessments, and clinical observations, but they have limitations such as scarce professional resources, test pressure affecting accuracy, and difficulty capturing complex symptoms with a single dimension.

This project integrates two complementary information sources: handwriting images and eye-tracking. Handwriting analysis: Dyslexic patients exhibit patterns like irregular letter shapes and chaotic stroke sequences. AI-driven image processing can quantify subtle abnormalities without being affected by subjective judgment. Eye-tracking: Patients show abnormal patterns such as longer fixation times and more regressions while reading; eye trackers record these precisely to provide diagnostic clues.

The technical architecture includes a handwriting image processing module (CNN for feature extraction + synthetic data to enhance generalization), an eye-tracking signal analysis module (RNN/Transformer to handle temporal dependencies), and a multimodal fusion layer (feature-level/decision-level fusion or attention mechanism). Machine learning uses a combination of classical and deep learning methods: structured features use SVM/random forests (interpretable), while high-dimensional unstructured data uses deep learning (strong feature learning) to balance performance and interpretability.

Application scenarios include large-scale screening in schools (quickly identifying students needing professional evaluation) and clinical auxiliary judgment (reducing subjective bias). Social value: Lowering the threshold of screening costs, enabling more families to access early identification services, timely intervention to improve learning trajectories, and reducing self-esteem damage and academic failure.

Challenges include data privacy (sensitive biological information needs strict protection) and model fairness (training data needs to cover different ages/languages/cultures). Positioning: The AI system is an auxiliary tool for professional evaluation, not a substitute; the final diagnosis is made by professionals.

In the future, more modalities (voice, EEG) can be integrated, child-friendly non-invasive devices can be developed, and cross-cultural validation datasets can be established. The development of large language models and multimodal foundation models will enhance generalization and few-shot learning capabilities, adapting to scenarios with limited data. This project provides an example of AI empowering education and health.