By means of an electron hole rate equation model we explain the phase dynamics of a quantum dot semiconductor optical amplifier and the appearance of different decay times observed in pump and probe experiments. The ultrafast hole relaxation leads to a first ultrafast recovery of the gain, followed by electron relaxation and, in the nanosecond timescale, radiative and non-radiative recombinations. The phase dynamics is slower and is affected by thermal redistribution of carriers within the dot. We explain the ultrafast response of quantum dot amplifiers as an effect of hole escape and recombination without the need to assume Auger processes.