Frequently, we use the Moore-Penrose pseudoinverse (MPP) even in cases when we do not require all of its defining properties. But if the running time and the storage size are critical, we can do better. By discarding some constraints needed for the MPP, we gain freedom to optimize other aspects of the new pseudoinverse. A sparser pseudoinverse reduces the amount of computation and storage. We propose a method to compute a sparse pseudoinverse and show that it offers sizable improvements in speed and storage, with a small loss in the least-squares performance. Differently from previous approaches, we do not attempt to approximate the MPP, but rather to produce an exact but sparse pseudoinverse. In the underdetermined (compressed sensing) scenario we prove that the rescaled sparse pseudoinverse yields an unbiased estimate of the unknown vector, and we demonstrate its potential in iterative sparse recovery algorithms, pointing out directions for future research.