Test time augmentation has been shown to be an effective approach to combat domain shifts in deep learning. Despite their promising performance levels, the interpretability of the underlying used models is however low. Saliency maps have been widely used in medical image analysis as a post-hoc interpretability method for deep learning models. Beyond explainability, in this study, we propose SaGTTA (Saliency Guided Test Time Augmentation), the first learnable framework that introduces saliency information to guide test time augmentations via a novel self-supervised loss term. During test time augmentation, the proposed self-supervised saliency-guided loss aims at promoting augmentation policies that enhance the distinctiveness among class-specific saliency maps. By promoting saliency distinctiveness among different labels of the test image during test time augmentation, the data distribution discrepancy between the test image and training dataset is alleviated. We compared the proposed method with a state-of-the-art method, using a publicly available dataset, showing improvements in terms of performance, model calibration, and robustness. The code will be made publicly available at https://github.com/yousuhang/SaGTTA.