In this supplementary material, we present the details of the neural network architecture and training settings used in all our experiments. This holds for all experiments presented in the main paper as well as in this supplementary material. We also show the summary results of all of our 96 experiments (test accuracy, training cross-entropy loss, and regularization loss), sampled at 100 epoch intervals. We analyze these results for each of the benchmark datasets, namely MNIST, Fashion-MNIST and CIFAR10, and underline global observations we make throughout the entire experiment set.