Brief

Researchers have developed a new two-stage framework for visual automatic speech recognition (V-ASR) that aims to improve accuracy by focusing on phonemes rather than direct word prediction. The system first fuses visual cues and facial landmark motion features to predict phonemes, then utilizes a large language model (LLM) called NLLB for word reconstruction. This approach reportedly achieves a 17.4% word error rate on the LRS2 dataset and 21.0% on LRS3, outperforming previous methods that struggled with viseme ambiguity. AI

Researchers have developed a new framework called HP-VSR-ResFiLM to improve visual speech recognition (VSR) by explicitly incorporating head-pose information. This method uses a pose-conditioned residual Feature-wise Linear Modulation (FiLM) block to adapt visual features based on head orientation, addressing challenges like geometric distortions and occlusions. Experiments on LRS2 and LRS3 datasets showed competitive performance, with word error rates of 25.0% and 33.2% respectively, demonstrating improved robustness for unconstrained VSR scenarios. AI