Transport of charged carriers in regimes of strong non-equilibrium is critical in a wide array of applications ranging from solar energy conversion and semiconductor devices to quantum information. Plasmonic hot-carrier science brings this regime of transport physics to the forefront since photo-excited carriers must be extracted far from equilibrium to harvest their energy efficiently. Here, we present a theoretical and computational framework, Non-Equilibrium Scattering in Space and Energy (NESSE), to predict the spatial evolution of carrier energy distributions that combines the best features of phase-space (Boltzmann) and particle-based (Monte Carlo) methods. Within the NESSE framework, we bridge first-principles electronic structure predictions of plasmon decay and carrier collision integrals at the atomic scale, with electromagnetic field simulations at the nano- to mesoscale. Finally, we apply NESSE to predict spatially-resolved energy distributions of photo-excited carriers that impact the surface of experimentally realizable plasmonic nanostructures, enabling first-principles design of hot carrier devices.