Class-specific classifiers for audio, visual and audio-visual speech recognition systems are developed and compared with traditional classifiers. We use state- of-the-art feature extraction methods and show the benefits of a class-specific classifier on each modality in speech recognition experiments.