Real-time Sign Language Interpretation Using Deep Learning

Abstract

Sign Language Serves As The Primary Mode Of Communication For Millions Of Hearing-impaired Individuals Worldwide, Yet The Gap Between The Deaf Community And The Hearing World Remains A Significant Barrier. This Paper Presents A Comprehensive, Three-subsystem Deep Learning Framework For Real-Time Sign Language Interpretation, Encompassing Sign-to-Text Conversion, Text-to-Animation Synthesis, And Text-to-Image Generation. The Sign-to-Text Module Leverages A Convolutional Neural Network (CNN) Trained On MediaPipe-extracted Hand Landmark Vectors, Deployed Via A Flask Backend With OpenCV-based Real-time Video Capture, Achieving A Classification Accuracy Of 97.6% Across 26 American Sign Language (ASL) Gestures. The Text-to-Animation Subsystem Utilizes Angular 21 And Ionic 8 On The Frontend With Node.js/Express And Firebase Cloud Infrastructure, Employing MediaPipe Holistic For Body-pose-driven Skeletal Animation Rendering. The Text-to-Image Subsystem Is A React 19 Single-page Application Powered By A Generative Deep Learning Model Pipeline, Converting Descriptive Text Into Contextual Sign Language Visual Representations. Extensive Experiments Demonstrate Strong Cross-subsystem Performance, With MSE Of 0.043, RMSE Of 0.207, Precision Of 96.8%, And Recall Of 97.1% On The Test Set. The Proposed Integrated Framework Significantly Advances Assistive Communication Technology And Establishes A Replicable Architecture For Real-world Sign Language Translation Deployment.

Keywords