This paper presents a unified semiclassical framework for five regimes of quantum fidelity decay and conjectures a new universal regime. The theory is based solely on the statistics of actions in the dephasing representation. Counterintuitively, in this representation, all of the decay is due to interference and none due to the decay of classical overlaps. Both rigorous and numerical support of the theory is provided.