To retrieve Surface Solar Radiation (SSR) from satellite images, a baseline reflectance of the observed ground in clear-sky conditions, also called background reflectance, is necessary to distinguish atmospheric absorption and scattering effects from surface reflective effects. Traditional physics-based methods approximate the background reflectance using the reflectance statistics of a rolling window over past observations. The underlying assumption is the higher dynamism of atmospheric conditions vs. ground reflectance properties. Inaccuracy increases in times and places of fast changing ground albedo. Modern machine-learning approaches have matched the accuracy of the physics-based methods, but also struggle to generalize to different region types without finetuning. Similarly, ground albedo was found to be a key modulator of SSR estimates generalization. We tested the hypothesis that a context-aware deep learning model would mimic the traditional strategy of leveraging the ground's albedo high inertia from past sequences and better estimate SSR, both in snow-free flat regions, and in mountainous regions with complex, snow-prone topography. Providing a temporal context, i.e. a window of past satellite observations, significantly improved ML-based SSR retrievals by enabling the model to implicitly reconstruct background reflectance.