Monitoring forests, in particular their response to climate and land use change, requires studying long time scales. While efficient deep learning methods have been developed to process short time series of satellite imagery, leveraging long time series of aerial imagery remains a challenge, due to changes in imaging technologies, sensors, and acquisition conditions, as well as irregular time gaps between acquisitions. In this work, we tackle this challenge through the task of multi -temporal forest mapping at the treeline ecotone in the Swiss Alps. We work with time series of aerial imagery spanning the years 1946-2020, without forest segmentation labels except for the year 2020. We propose a multi -temporal deep learning method which takes irregular time gaps into consideration, and learns to overcome the large domain shift between acquisitions through a custom loss function encoding prior knowledge about forest cover dynamics. Using this method, we significantly improve the forest segmentation performance on historical images compared to a mono -temporal counterpart processing each acquisition independently, or multi -temporal counterparts trained with more generic temporal consistency loss functions. We show that our method is a promising approach for monitoring subtle temporal trends from heterogeneous remote sensing time series. Overall, our work suggests that by designing a deep learning architecture and a training procedure based on problem -specific prior knowledge, a variety of Earth processes can be monitored from long time series of remote sensing data with incomplete training labels.