Plasmonic hot carrier devices extract excited carriers from metal nanostructures before equilibration and have the potential to surpass semiconductor light absorbers. However their efficiencies have so far remained well below theoretical limits, which necessitates quantitative prediction of carrier transport and energy loss in plasmonic structures to identify and overcome bottlenecks in carrier harvesting. Here, we present a theoretical and computational framework, nonequilibrium scattering in space and energy (NESSE), to predict the spatial evolution of carrier energy distributions that combines the best features of phase-space (Boltzmann) and particle-based (Monte Carlo) methods. Within the NESSE framework, we bridge first-principles electronic structure predictions of plasmon decay and carrier collision integrals at the atomic scale, with electromagnetic field simulations at the nano- to mesoscale. Finally, we apply NESSE to predict spatially-resolved energy distributions of photoexcited carriers that impact the surface of experimentally realizable plasmonic nanostructures at length scales ranging from tens to several hundreds of nanometers, enabling first-principles design of hot carrier devices.