Landau damping of coherent modes is strongly dependent on the exact shape of the particle bunches. One often assumes that the transverse distributions in high-energy hadron colliders can be approximated by Gaussian distributions, in acceptable agreement with measurements, but known to be only a first approximation. In this paper, it is investigated how a specific change of the transverse distributions can cause a loss of Landau damping. A mechanism is introduced where the coherent modes, which are excited by noise in the machine, act back on the individual particles through wakefields. The impact is modeled as a narrow diffusion in frequency space, and therefore also in action space due to amplitude dependent detuning, which leads to a local flattening of the distribution. This distribution evolution corresponds to the drilling of a borehole in the stability diagram, i.e. a local reduction of the imaginary part of the curve. Hence, initially stable regions are changed into unstable ones at the real frequencies of the coherent modes. To mitigate this instability mechanism, one must operate the machine with a stability margin of magnitude that depends on the noise amplitude and the coherent modes. In this model, the latency is defined as the time from the start of the noise excitation, on an initially Gaussian distributed bunch, to the bunch instability. The proposed model is found to agree with results in dedicated latency experiments performed in the LHC, where bunches eventually went unstable with more than twice the detuning strength required for the stabilization of a Gaussian distribution.