Marginal stability is the notion that stability is achieved, but only barely so. This property constrains the ensemble of configurations explored at low temperature in a variety of systems, including spin, electron, and structural glasses. A key feature of marginal states is a (saturated) pseudogap in the distribution of soft excitations. We examine how such pseudogaps appear dynamically by studying the Sherrington-Kirkpatrick (SK) spin glass. After revisiting and correcting the multi-spin-flip criterion for local stability, we show that stationarity along the hysteresis loop requires soft spins to be frustrated among each other, with a correlation diverging as C(λ) ∼ 1/λ, where λ is the stability of the more stable spin. We explain how this arises spontaneously in a marginal system and develop an analogy between the spin dynamics in the SK model and random walks in two dimensions. We discuss analogous frustrations among soft excitations in short range glasses and how to detect them experimentally. We also show how these findings apply to hard sphere packings. © 2015 American Physical Society.