Machine Learning techniques play an increasingly vital role in the analysis of Biomedical imagery, as in all other areas of Computer Vision. However, in this specific context, they suffer from the fact that experimental conditions and protocols change often and that acquiring sufficient amounts of new training data after each image acquisition is impractical. In this paper, we propose an effective method to train a non-linear SVM using a very small amount of new data by leveraging data obtained under different conditions. Unlike earlier approaches, ours takes full advantage of the kernelized SVM formulation, does not depend on a loss function that is sensitive to outliers, and yields a quadratic optimization problem. We demonstrate its effectiveness for the purpose of classifying pixels in electron microscope image stacks and delineating linear structures in optical microscopy and retinal scans. Our method outperforms two state-of-the-art transfer-learning approaches in terms of accuracy and computational complexity.