This paper studies the rate distortion behavior of sparse memoryless sources that serve as models of sparse signal representations. For the Hamming distortion criterion, $R(D)$ is shown to be essentially linear. For the mean squared error measure, two models are analyzed: the mixed discrete/continuous spike processes and Gaussian mixtures. The latter are shown to be a better model for ``natural'' data such as sparse wavelet coefficients. Finally, the geometric mean of a continuous random variable is introduced as a sparseness measure. It yields upper and lower bounds on the entropy and thus characterizes high-rate $R(D)$.