AL2: Progressive Activation Loss for Learning General Representations in Classification Neural Networks

The large capacity of neural networks enables them to learn complex functions. To avoid overfitting, networks however require a lot of training data that can be expensive and time-consuming to collect. A common practical approach to attenuate overfitting is the use of network regularization techniques. We propose a novel regularization method that progressively penalizes the magnitude of activations during training. The combined activation signals produced by all neurons in a given layer form the representation of the input image in that feature space. We propose to regularize this representation in the last feature layer before classification layers. Our method's effect on generalization is analyzed with label randomization tests and cumulative ablations. Experimental results show the advantages of our approach in comparison with commonly-used regularizers on standard benchmark datasets.


Published in:
[Proceedings of ICASSP 2020]
Presented at:
45th International Conference on Acoustics, Speech, and Signal Processing - ICASSP 2020, Barcelona, Spain, 4-6 May, 2020
Year:
2020
Publisher:
IEEE
Keywords:
Note:
All papers accepted to ICASSP 2020 will be published on IEEE Xplore through Open Preview on 9 April 2020, and will be freely accessible and downloadable by all in final format from 9 April to 8 May 2020.
Laboratories:


Note: The status of this file is: Anyone


 Record created 2020-02-13, last modified 2020-10-25

Fulltext:
Download fulltext
PDF

Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)