Real-time AI Sign Language Translation System: An American Sign Language Recognition Solution Based on MediaPipe and Deep Learning

This project aims to break the communication barrier between the hearing-impaired and hearing people, presenting a complete open-source real-time sign language recognition system. The system combines MediaPipe hand key point detection, TensorFlow/Keras neural networks, and ensemble learning methods to achieve real-time recognition of American Sign Language (ASL) static letter gestures from a camera, and supports text-to-speech output. The project is maintained by harunhuskic and open-sourced on GitHub (link: https://github.com/harunhuskic/Real-Time-AI-Sign-Language-Interpreter), released on June 1, 2026.

Sign language is the main communication method for the hearing-impaired, but most hearing people are not familiar with it, causing communication barriers. Traditional recognition methods based on data gloves are accurate but expensive and inconvenient to wear; contactless solutions based on computer vision are easier to promote but face challenges such as rapid gesture changes, lighting differences, varying user hand shapes, and performance requirements for real-time processing. American Sign Language includes 26 letter gestures, some static (e.g., A, B, C) and some with motion trajectories (e.g., J, Z). This project focuses on static letter recognition, laying the foundation for dynamic gesture expansion.

The system adopts a modular design, including five major components:

1. Data Collection and Preprocessing: Supports users to collect gesture samples via camera, automatically extracts and annotates hand images, facilitating scene optimization and expansion to other sign language systems.
2. Feature Extraction: Uses MediaPipe to detect 21 hand key points (including wrist, finger joints, etc.). The key point coordinates have geometric invariance, strong robustness, and reduce input dimensionality.
3. Neural Network Recognition: Builds a model based on TensorFlow/Keras. The input is 42-dimensional normalized coordinates (x and y of 21 points), processed through fully connected layers + Dropout regularization, outputting a probability distribution of 26 classes; uses an ensemble learning voting mechanism to improve stability.
4. Real-time Inference Post-processing: Improves user experience through voting buffer (confirmation only when consecutive frames are consistent), confidence threshold (filtering noise), and word buffer control (supports word combination and editing).
5. Voice Output: The recognition results are broadcasted via a text-to-speech engine, completing the translation link from sign language to voice.

MediaPipe Integration

MediaPipe Hands uses a two-stage architecture: first locates the palm area, then regresses the coordinates of 21 key points, achieving real-time performance of >30 FPS on mobile devices. The key points include 1 wrist, 4 thumb points (from metacarpophalangeal joint to fingertip), and 4 points for each of the other four fingers.

Neural Network Design

Lightweight architecture: Input layer (42) → Hidden layer 1 (128, ReLU) → Dropout (0.2) → Hidden layer 2 (64, ReLU) → Dropout (0.2) → Output layer (26, Softmax). Uses categorical cross-entropy loss and Adam optimizer to balance expressive power and complexity.

Ensemble Learning Strategy

Trains multiple independent models, improves prediction stability through majority voting mechanism, and outputs results only when most models agree and the confidence is sufficient.

Immediate Application Scenarios

1. Personal auxiliary tool: Converts sign language to text/voice for hearing-impaired users in daily communication;
2. Education and training: Helps sign language learners correct gestures and provides immediate feedback;
3. Public service windows: Deployed in banks, hospitals, etc., to improve barrier-free service levels.

Future Expansion Directions

1. Dynamic gesture support: Expand recognition to gestures with motion trajectories (e.g., J, Z) and complete vocabulary;
2. Multilingual sign language: Adapt to different systems such as Chinese Sign Language, British Sign Language;
3. Bilateral gesture recognition: Support sign language with both hands;
4. Mobile deployment: Optimize model size and computation, develop iOS/Android applications.

This project integrates MediaPipe (reliable key point detection), TensorFlow/Keras (flexible modeling), ensemble learning (robustness), and modular design (easy expansion) to form a complete application system. For entry-level developers, it is an excellent reference project covering data collection, model training, and deployment. Its social value is significant: technology should benefit the hearing-impaired group and promote barrier-free communication. The open-source feature allows global developers to jointly improve it, adapt to more sign language systems, and promote technology popularization.