Accurate retrieval of surface solar radiation (SSR) from satellite imagery critically depends on estimating the background reflectance that a spaceborne sensor would observe under clear-sky conditions. Deviations from this baseline can then be used to detect cloud presence and guide radiative transfer models in inferring atmospheric attenuation. Operational retrieval algorithms typically approximate background reflectance using monthly statistics, assuming surface properties vary slowly relative to atmospheric conditions. Here, we show that deep-learning models also benefit from past temporal context for retrieving SSR by implicitly learning to infer clear-sky surface reflectance from raw satellite image sequences. We propose an attention-based emulator for SSR retrieval and show it matches the performance of albedo-informed models, when provided a sufficiently long temporal context. The emulator is trained on instantaneous SSR estimates over Switzerland, a region characterized by complex terrain and dynamic snow cover. Our geospatial analysis shows that mountainous regions benefit most from providing a larger temporal context to the model. Root-mean-square deviations from ground truth of 134.0 W/m2 for instantaneous measurements, 52.9 W/m2 and 22.5 W/m2 for daily and monthly aggregates respectively, are achieved with a substantial computational speedup over the original pipeline.