We present a single-photon-sensitive spectrometer based on a linear array of 512 single-photon avalanche diode detectors with 0.04 nm spectral and 40 ps temporal resolutions. We employ a fast data-driven operation that allows direct measurement of time and frequency for simultaneous single photons, time- and frequency-stamping each single-photon detection. Our results combine excellent temporal and spectral resolution. This work opens numerous applications in quantum photonics, especially when both spectral and temporal properties of single photons can be exploited.