We consider the rate-distortion problem for non-bandlimited signals that have a finite rate of innovation, in particular, the class of continuous periodic stream of Diracs, characterized by a set of time positions and weights. Previous research has only considered the sampling of these signals, ignoring quantization, which is necessary for any practical application (e.g. UWB, CDMA). In order to achieve accuracy under quantization, we introduce two types of oversampling, namely, oversampling in frequency and oversampling in time. The reconstruction accuracy is measured by the MSE of the time positions. High accuracy is achieved by enforcing the reconstruction to satisfy either three convex sets of constraints related to: 1) sampling kernel, 2) quantization and 3) periodic streams of Diracs, which is then said to provide strong consistency or only the first two, providing weak consistency. We propose reconstruction algorithms for both weak and strong consistency. Regarding the rate, we also consider a threshold crossing based scheme, which is more efficient than the PCM encoding. We compare the rate- distortion behavior that is obtained from both increasing the oversampling in time and in frequency, on the one hand, and, on the other hand, from decreasing the quantization stepsize.