Measuring the effect of nuisance variables on classifiers

In real-world classification problems, nuisance variables can cause wild variability in the data. Nuisance corresponds for example to geometric distortions of the image, occlusions, illumination changes or any other deformations that do not alter the ground truth label of the image. It is therefore crucial that designed classifiers are robust to nuisance variables, especially when these are deployed in real and possibly hostile environments. We propose in this paper a probabilistic framework for efficiently estimating the robustness of state-of-the-art classifiers and sampling problematic samples from the nuisance space. This allows us to visualize and understand the regions of the nuisance space that cause misclassification, in the perspective of improving robustness. Our probabilistic framework is applicable to arbitrary classifiers and potentially high-dimensional and complex nuisance spaces. We illustrate the proposed approach on several classification problems and compare classifiers in terms of their robustness to nuisances. Moreover, using our sampling technique, we visualize problematic regions in the nuisance space and infer insights into the weaknesses of classifiers as well as the features used in classification (e.g., in face recognition). We believe the proposed analysis tools represent an important step towards understanding large modern classification architectures and building architectures with better robustness to nuisance.

Presented at:
British Machine Vision Conference (BMVC), York, UK, September 19-22, 2016

 Record created 2016-08-06, last modified 2019-08-12

final332 - Download fulltextPDF
abstract332 - Download fulltextPDF
supp332 - Download fulltextPDF
Rate this document:

Rate this document:
(Not yet reviewed)