Neural Anisotropy Directions

In this work, we analyze the role of the network architecture in shaping the inductive bias of deep classifiers. To that end, we start by focusing on a very simple problem, i.e., classifying a class of linearly separable distributions, and show that, depending on the direction of the discriminative feature of the distribution, many state-of-the-art deep convolutional neural networks (CNNs) have a surprisingly hard time solving this simple task. We then define as neural anisotropy directions (NADs) the vectors that encapsulate the directional inductive bias of an architecture. These vectors, which are specific for each architecture and hence act as a signature, encode the preference of a network to separate the input data based on some particular features. We provide an efficient method to identify NADs for several CNN architectures and thus reveal their directional inductive biases. Furthermore, we show that, for the CIFAR-10 dataset, NADs characterize features used by CNNs to discriminate between different classes.


Published in:
Proceedings of the Neural Information Processing Systems Conference
Presented at:
NeurIPS 2020 34th Conference on Neural Information Processing Systems, Vancouver, Canada, December 6-12, 2020
Year:
2020
Keywords:
Laboratories:


Note: The status of this file is: Anyone


 Record created 2020-09-29, last modified 2020-10-29

PREPRINT:
Download fulltext
PDF

Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)