The electronic, optical, and magnetic properties of quantum solids are determined by their low-energy (<100 meV) many-body excitations. Dynamical characterization and manipulation of such excitations rely on tools that combine nm-spatial, fs-temporal, and meV-spectral resolution. Currently, phonons and collective plasmon resonances can be imaged in nanostructures with atomic (sub-nm) and tens of meV space/energy resolution using state-of-the-art energy-filtered transmission electron microscopy (TEM), but only under static conditions, while fs-resolved measurements are common but lack spatial or energy resolution. Here, we demonstrate a new method of spectrally resolved photon-induced near-field electron microscopy (SRPINEM) that allows us to obtain nm-fs-resolved maps of nanoparticle plasmons with an energy resolution determined by the laser line width (20 meV in this work) and no longer limited by the electron beam and spectrometer energy spreading. This technique can be extended to any optically accessible low-energy mode, thus pushing TEM to a previously unattainable spectral domain with an unprecedented combination of space, energy, and temporal resolution.