Deep Learning-Based Audio-Visual Speech Recognition for Bosnian Digits

• Published in: JURNAL KEJURUTERAAN
• Main Authors: Fazlic, Husein; Abd Almisre, Ali; Tahir, Nooritawati Md
• Format: Article
• Language: English
• Published: UKM PRESS 2024
• Subjects: Engineering
• Online Access: https://www-webofscience-com.uitm.idm.oclc.org/wos/woscc/full-record/WOS:001157147500024
• ISSN: 0128-0198; 2289-7526
• DOI: 10.17576/jkukm-2024-36(1)-14

This study presents a deep learning-based solution for audio-visual speech recognition of Bosnian digits. The task posed a challenge due to the lack of an appropriate Bosnian language dataset, and this study outlines the approach to building a new dataset. The proposed solution includes two components: visual speech recognition, which involves lip reading, and audio speech recognition. For visual speech recognition, a combined CNN-RNN architecture was utilised, consisting of two CNN variants namely Google Net and ResNet-50. These architectures were compared based on their performance, with ResNet-50 achieving 72% accuracy and Google Net achieving 63% accuracy. The RNN component used LSTM. For audio speech recognition, FFT is applied to obtain spectrograms from the input speech signal, which are then classified using a CNN architecture. This component achieved an accuracy of 100%. The dataset was split into three parts namely for training, validation, and testing purposes such that 80%, 10% and 10% of data is allocated to each part, respectively. Furthermore, the predictions from the visual and audio models were combined that yielded 100% accuracy based on the developed dataset. The findings from this study demonstrate that deep learning-based methods show promising results for audio-visual speech recognition of Bosnian digits, despite the challenge of limited Bosnian language datasets.