While learned image features can achieve great accuracy on different Computer Vision problems, their use in real-world situations is still very limited as their extraction is typically time-consuming. We therefore propose a method to learn image features that can be extracted very efficiently using separable filters, by looking for low rank filters. We evaluate our approach on both the image categorization and the pixel classification tasks and show that we obtain similar accuracy as state-of-the-art methods, at a fraction of the computational cost.