Gels made of telechelic polymers connected by reversible cross-linkers are a versatile design platform for biocompatible viscoelastic materials. Their linear response to a step strain displays a fast, near-exponential relaxation when using low-valence cross-linkers, while larger supramolecular cross-linkers bring about much slower dynamics involving a wide distribution of timescales whose physical origin is still debated. Here, we propose a model where the relaxation of polymer gels in the dilute regime originates from elementary events in which the bonds connecting two neighboring cross-linkers all disconnect. Larger cross-linkers allow for a greater average number of bonds connecting them but also generate more heterogeneity. We characterize the resulting distribution of relaxation timescales analytically and accurately reproduce stress relaxation measurements on metal-coordinated hydrogels with a variety of cross-linker sizes including ions, metal-organic cages, and nanoparticles. Our approach is simple enough to be extended to any cross-linker size and could thus be harnessed for the rational design of complex viscoelastic materials.