Projects

Neural Network Speech Recognition

Abstract

This work focuses on the research related to enabling individuals with speech impairment to use speech-to-text software to recognize and dictate their speech. Automatic Speech Recognition (ASR) tends to be a challenging problem for researchers because of the wide range of speech variability. Some of the variabilities include different accents, pronunciations, speeds, volumes, etc. It is very difficult to train an end-to-end speech recognition model on data with speech impediment due to the lack of large enough datasets, and the difficulty of generalizing a speech disorder pattern on all users with speech impediments. This work highlights the different techniques used in deep learning to achieve ASR and how it can be modified to recognize and dictate speech from individuals with speech impediments.

Networks Architecture and Edit Distance

The project is split into three consecutive processes; ASR to phonetic transcription, edit distance and language model. The ASR is the most challenging due to the complexity of the neural network architecture and the preprocessing involved. We apply Mel-Frequency Cepstrum Coefficients (MFCC) to each audio file which results in 13 coefficients for each frame. The labels (text matching the audio) is converted to phonemes using the CMU arpabet phonetic dictionary. The Network is trained using the MFCC coefficients as inputs and phonemes' IDs as outputs. The Network architecture implemented is a Bidirectional Recurrent Deep Neural Network (BRDNN – fig.1), it consists of 2 LSTM cells (one in each direction) with 100 hidden blocks in each direction. The network is made deep by stacking two more layers, which results in a 3 layers network in depth. Two fully connected layers were attached to the output of the recurrent network with 128 hidden units in each. This architecture resulted in a 38.5% LER on the Test set.

Figure 1: Deep Bidirectional LSTM

Levenshtein edit distance (fig. 2) is used to generate potential words from phonemes. Edit distance of one means a maximum change of one phoneme is allowed, edit distance of two means a change of one or two phonemes is allowed when generating the potential words, and so on. These changes can be inserts, deletes or replacements. The language model uses the potential words to generate sentences with the most semantic meaning. The language model is another recurrent neural network model trained on full sentences. The model outputs the probability of a word occurring after a given word or sentence. It is simpler than the main speech recognition model because it is not bidirectional and not as deep. The language model uses beam search decoding to find the best sentences.

Figure 2: A) Edit operations B) Dynamic programming of Edit Distance C) Algorithm from Wikipedia