Real-time Indian Sign Language (isl) Recognition And Multilingual Translation System Using Deep Learning And Natural Language Processing

Abstract

Artificial Intelligence (AI) Has Become A Powerful Tool In Developing Assistive Technologies That Improve Accessibility For Individuals With Disabilities. Among These, Hearing And Speech-impaired Individuals Face Significant Challenges In Communication Due To The Lack Of Widespread Understanding Of Indian Sign Language (ISL). ISL Is A Visual Language That Relies On Gestures, Facial Expressions, And Body Movements. However, Most People Are Not Familiar With It, Leading To Communication Barriers In Everyday Life. Existing Systems Mainly Focus On Recognizing Individual Alphabets Or Static Gestures, Which Limits Their Ability To Provide Real-time, Meaningful Communication. This Paper Proposes An AI-based Two-way Communication System Designed To Bridge The Gap Between Hearing-impaired Individuals And Others. The System Converts ISL Gestures Into Text And Speech While Also Translating Spoken Language Into Text, Enabling Bidirectional Communication. The Proposed Approach Integrates Computer Vision, Machine Learning, And Natural Language Processing Techniques. MediaPipe Is Used For Capturing Real-time Hand Landmarks, While Long Short-Term Memory (LSTM) Networks Are Employed For Recognizing Dynamic Gesture Sequences. Recognized Gestures Are Mapped Into Gloss Representations, Which Are Further Processed Into Meaningful Sentences Using NLP Models Such As BART. Additionally, Multilingual Translation Is Achieved Using IndicTrans2, And Speech Output Is Generated Using Indic Text-to-Speech Systems. The System Is Designed To Be Efficient, Cost-effective, And User-friendly, Making It Suitable For Real-world Applications. The Results Demonstrate Improved Accuracy, Real-time Performance, And Better Contextual Understanding Compared To Existing Methods. This Research Contributes To Enhancing Accessibility And Promoting Inclusivity By Enabling Effective Communication Between Hearing And Non-hearing Communities.

Keywords