The capability to learn from experience is a key property for a visual recognition algorithm working in realistic settings. This paper presents an SVM-based algorithm, capable of learning model representations incrementally while keeping under control memory requirements. We combine an incremental extension of SVMs with a method reducing the number of support vectors needed to build the decision function without any loss in performance, introducing a parameter which permits a user-set trade-off between performance and memory. The resulting algorithm is guaranteed to achieve the same recognition results as the original incremental method while reducing the memory growth. Moreover, experiments in two domains of material and place recognition show the possibility of a consistent reduction of memory requirements with only a moderate loss in performance. For example, results show that when the user accepts a reduction in recognition rate of 5%, this yields a memory reduction of up to 50%.