To operate the railway system safely and efficiently, a multitude of assets need to me monitored. Railway sleepers are one of these infrastructure assets, that are safety critical. To automate the monitoring process, data-driven fault diagnostics models have shown great potential. However, in practice, the performance of data-driven models can be compromised if the training dataset is not representative of all possible future conditions. Environmental factors, for example, can continuously change and cause variations in the condition monitoring data. The caused variations can harm the performance of a data-driven diagnostics models if (1) they are not represented in the training datatset and especially, if (2) they are larger compared to variations caused by a change in the health condition. We propose to approach this problem by learning a feature representation that is, on the one hand, invariant to operating or environmental factors but, on the other hand, sensitive to changes in the asset's health condition. We evaluate how contrastive learning can be employed to tackle this challenge on a supervised diagnostics task given a real condition monitoring dataset of railway sleepers. We evaluate the performance of supervised contrastive feature learning given a labeled image dataset that is collected by a diagnostic vehicle. Our results demonstrate that contrastive feature learning significantly improves the performance on the supervised classification task regarding sleepers compared to a state-of-the-art method.