Recent years have seen an increasing interest in sparseness constraints for image classification and object recognition, probably motivated by the evidence of sparse representations internal in the primate visual cortex. It is still unclear, however, whether or not sparsity helps classification. In this paper we analyze the image classification task on CIFAR-10, a very challenging dataset, and evaluate the impact of sparseness on the recognition rate using both standard and learned filter banks in a modular architecture. In our experiments, enforcing sparsity constraints is not required at run-time, since no performance improvement has been reported by indiscriminatively sparsifying the descriptors. This observation complies with the most recent findings on human visual cortex suggesting that a feed-forward mechanism underlies object recognition, and is of practical interest, as enforcing these constraints can have a heavy computational cost. Our best method outperforms the state-of-the-art, from 64.84% success rate on color images to 71.53% on grayscale images.