Context & Problem
Visual speech recognition (lip reading) is a challenging task in assistive technology and noisy environments. LipSense translates video sequences of lip movements to text using PyTorch, resolving spatial and temporal cues without audio feeds.
Architecture
System pipeline layout representing coordination blocks and execution pathways designed for this solution.
Implementation
- Designed a 3D Convolutional Neural Network (3D-CNN) to extract spatiotemporal visual features from lip movements.
- Wired a Bidirectional GRU network to decode frame sequences into character probability matrices.
- Tuned Connectionist Temporal Classification (CTC) Loss function parameters to train the model end-to-end without frame alignments.
Challenges & Trade-offs
CHALLENGE: Drastic variations in lighting and speaker head orientations.
RESOLUTION: Implemented a dynamic face-alignment step using MediaPipe landmarks, normalising coordinates and bounding box scales around the lips ROI prior to classification.
Results & Impact
Quantitative metrics and verified outcomes obtained from benchmarks.
95.2%Transcription Accuracy
4.8%Word Error Rate (WER)
30fpsExtraction Speed
Media Gallery
Dashboard screens and coordinate visualizations of the running system.
Visual landmark extraction nodes mapped around the lip ROI