Sign Language Detection

Abstract

1. Introduction

EchoSign This gif shows ASL sign detetcion and recognition of the class which is written as interpreted text.To hear the text-to-speech, navigate to Live_ASL_detection_TTS.mp4

2. Objective

EchoSign was made as part of an IBM internship project which we won with this project. It uses transfer learning on MobileNet on a hand-curated dataset of ASL images. The website for this classification was developed in Flask and it uses TTS technology for ASL text to speech conversion.

This project demonstrates how HTML can be applied to a real-world AI & Machine Learning problem.

3. Key Features / Modules

• This website was developed using Python, Flask, HTML, CSS & Javascript.
• Users who wish to converse in ASL, may use this detection system to convert the interpreted text to speech for communicating without bounds.
• Users who aren't equipped with ASL may use the English to ASL converter to learn relevant symbols.
• The voice for TTS technology may be changed in settings.
• Developers may check the raw footage opened using SocketIO and the prediction on top of that OpenCV feed.
• Text-to-speech translation uses SpeechSynthesisUtterance() of the Web Speech API.
• TensorFlow 2.0 and Keras library is used in the development of our model. The final model used for testing and deployment is MobileNet.
• Other machine learning techniques employed are:
• Data Augmentation
• Transfer Learning

4. Technology Stack

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

• A modern web browser
• VS Code or any code editor
• Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/priyanka-maz/sign-language-detection.git
cd sign-language-detection

1. Click on 'Start' on the homepage to begin sign language detection on your video feed
2. Permissions for video feed usage must be set to 'Allow'
3. Write letters in American Sign Language to form words or phrases which show up on the console on the right
4. Click on 'Play' button below the console to convert your sentences from text-to-speech for seamless communication
5. For non-ASL users, you can refer to the English to ASL converter on the homepage
6. Follow 'Tips' under console for further instructions

python3 -m venv /path/to/venv
cd /path/to/venv
git clone https://github.com/priyanka-maz sign-language-detection
source bin/activate
cd sign-language-detection
python3 -m pip install -r requirements.txt
python3 app.py

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

• Deploy the model as a web app with Streamlit, Flask or FastAPI
• Compare against an additional model and report the metric difference
• Add explainability (SHAP / Grad-CAM)

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

1. What dataset does the project use and how was it pre-processed?
2. Which algorithm / model architecture is used and why was it chosen over alternatives?
3. How are training and testing data split, and how is overfitting avoided?
4. Which evaluation metrics (accuracy, precision, recall, F1) are reported and what do they mean here?
5. How would you deploy this model for real users?

9. Source Code & License

This project is developed by priyanka-maz and published on GitHub under the MIT License. Please follow the license terms and credit the original author when you use or modify this code.