Repository logo

Infoscience

  • English
  • French
Log In
Logo EPFL, École polytechnique fédérale de Lausanne

Infoscience

  • English
  • French
Log In
  1. Home
  2. Academic and Research Output
  3. Conferences, Workshops, Symposiums, and Seminars
  4. Combining Multiple Views for Visual Speech Recognition
 
conference paper

Combining Multiple Views for Visual Speech Recognition

Zimmermann, Marina  
•
Mehdipour Ghazi, Mostafa
•
Ekenel, Hazim Kemal
Show more
Ouni, Slim
•
Davis, Chris
Show more
2017
Proceedings of the 14th International Conference on Auditory-Visual Speech Processing (AVSP2017)
14th International Conference on Auditory-Visual Speech Processing (AVSP2017)

Visual speech recognition is a challenging research problem with a particular practical application of aiding audio speech recognition in noisy scenarios. Multiple camera setups can be beneficial for the visual speech recognition systems in terms of improved performance and robustness. In this paper, we explore this aspect and provide a comprehensive study on combining multiple views for visual speech recognition. The thorough analysis covers fusion of all possible view angle combinations both at feature level and decision level. The employed visual speech recognition system in this study extracts features through a PCA-based convolutional neural network, followed by an LSTM network. Finally, these features are processed in a tandem system, being fed into a GMM-HMM scheme. The decision fusion acts after this point by combining the Viterbi path log-likelihoods. The results show that the complementary information contained in recordings from different view angles improves the results significantly. For example, the sentence correctness on the test set is increased from 76% for the highest performing single view (30 degrees) to up to 83% when combining this view with the frontal and (60 degrees) view angles

  • Files
  • Details
  • Metrics
Loading...
Thumbnail Image
Name

Zimmermann_avsp2017.pdf

Access type

openaccess

Size

250 KB

Format

Adobe PDF

Checksum (MD5)

217c7c5087ccd43893bb31524c0773e0

Logo EPFL, École polytechnique fédérale de Lausanne
  • Contact
  • infoscience@epfl.ch

  • Follow us on Facebook
  • Follow us on Instagram
  • Follow us on LinkedIn
  • Follow us on X
  • Follow us on Youtube
AccessibilityLegal noticePrivacy policyCookie settingsEnd User AgreementGet helpFeedback

Infoscience is a service managed and provided by the Library and IT Services of EPFL. © EPFL, tous droits réservés