The (sub)-ps relaxation processes in PbSe quantum dots of different sizes were measured by near-IR luminescence upconversion. The early-time spectra are characterized by emission from several low-lying excited states. The direct measurement of the Huang-Rhys factors of the lowest excited states excludes strong electron-phonon coupling in the intraband relaxation. The kinetics point to a fast sequential cascade process between excited states, governed by energy gaps and a quasicontinuous valence-band structure.