Hyperspectral (HS) images are commonly used in the context of planetary exploration, particularly for the analysis of the composition of planets. As several instruments have been sent throughout the Solar System, a huge quantity of data is getting available for the research community. Among classical problems in the analysis of HS images, a crucial one is unsupervised nonlinear spectral unmixing, which aims at estimating the spectral signatures of elementary materials and determining their relative contribution at a subpixel level. While the unmixing problem is well studied for Earth observation, some of the traditional problems encountered with Earth images are somehow magnified in planetary exploration. Among them, large image sizes, strong nonlinearities in the mixing (often different from those found in the Earth images), and the presence of anomalies are usually impairing the unmixing algorithms. This paper presents a new method that scales favorably with the problem posed by this analysis. It performs an unsupervised unmixing jointly with anomaly-detection capacities and has a global linear complexity. Nonlinearities are handled by decomposing the HS data on an overcomplete set of spectra, combined with a specific sparse projection, which guarantees the interpretability of the analysis. A theoretical study is proposed on synthetic data sets, and results are presented over the challenging 4-Vesta asteroid data set.