Context. To deepen our understanding of the role of small-scale magnetic fields in active regions (ARs) and in the quiet Sun (QS) on the solar irradiance, it is fundamental to investigate the physical processes underlying their continuum brightness. Previous results showed that magnetic elements in the QS reach larger continuum intensities than in ARs at disk center, but left this difference unexplained.