Real-time Gesture Recognition System Based on LSTM: Enabling Machines to Understand Sign Language

This article introduces an open-source project completed by a computer science graduate from the University of Plymouth—a real-time American Sign Language (ASL) detection and translation system based on LSTM neural networks and MediaPipe human pose estimation technology. The system aims to bridge the communication gap between the hearing-impaired and hearing people, and discusses its technical principles and application prospects.

Approximately 70 million hearing-impaired people worldwide use sign language as their primary means of communication, but most hearing people do not understand sign language, leading to communication barriers. Traditional human translation is costly and difficult to popularize. The development of deep learning technology provides a new direction for solving this problem. This project, based on this background, transforms academic achievements into practical assistive technology.

Core Component: LSTM Neural Network

LSTM is a recurrent neural network suitable for processing sequence data. Through its gating mechanism, it captures the temporal dependencies of gesture movements and distinguishes similar gestures. Unlike CNNs which only process single frames, LSTM can consider the evolution of movements across multiple frames.

Pose Estimation: MediaPipe Framework

Google's open-source MediaPipe extracts coordinates of 21 hand joints, converting image data into low-dimensional feature vectors, reducing input dimensionality while ensuring real-time performance (30+ FPS on mobile devices).

Data Flow Process

Camera captures video stream → MediaPipe detects hand key points frame by frame to generate coordinate sequences → LSTM receives a fixed time window (e.g., 30 frames) to predict sign language vocabulary → Output text results.

Challenge 1: Real-time Requirements

Optimize performance through lightweight MediaPipe models, efficient LSTM architecture, and frame sampling strategies to ensure smooth communication.

Challenge 2: Gesture Diversity and Ambiguity

Use LSTM's sequence modeling capability to handle variable-length patterns, and may adopt data augmentation techniques (random scaling, time warping) to improve generalization.

Challenge 3: Continuous Sign Language Sentence Segmentation

Although focusing on word-level recognition, to support continuous translation, sliding windows combined with confidence thresholds may be introduced to determine vocabulary boundaries.

Education Sector

Assist communication between hearing-impaired children and their relatives; help hearing people learn sign language with instant feedback.

Public Services

Deploy in places like banks and hospitals to lower the threshold for hearing-impaired people to access services and enhance the inclusiveness of public services.

Remote Communication

Integrate with video conferencing platforms to enable hearing-impaired people to participate in remote work and online education without barriers.

Technical Limitations

Currently only supports ASL word-level recognition, lacking elements like grammatical structure and facial expressions; sign language has large regional differences (e.g., CSL vs ASL), so cross-language migration requires retraining.

Future Prospects

Introduce Transformer to replace LSTM; integrate facial expressions and upper body posture; build an end-to-end continuous sign language translation system; localize to adapt to specific sign language variants (e.g., Chinese Sign Language).

This project demonstrates the great potential of deep learning in the field of assistive technology and is a solid step towards barrier-free communication. With model optimization and reduced hardware costs, we look forward to 'machines understanding sign language' moving from the laboratory to daily life, becoming a bridge for communication among the hearing-impaired community.