The dynamic compression responses of dry quartz sand are tested with a modified spilt Hopkinson pressure bar (MSHPB), and the quasi-static compression responses are tested for comparison with a material testing system. In the experiments, the axial stress-strain responses and the confining pressure of the jacket are both measured. Comparison of the dynamic and the quasi-static axial stress-strain curves indicate that dry quartz sand exhibits obvious strain-rate effects. The grain size distributions of the samples after dynamic and quasi-static loading are obtained with the laser diffractometry technique to interpret the rate effects. Quantitative analyses of the grain size distributions show that at the same stress level, the particle breakage extent under quasi-static loading is larger than that under dynamic loading. Moreover, the experimental and the theoretical relationships of the particle breakage extent versus the plastic work show that the energy efficiency in particle breakage is higher under quasi-static loading, which is the intrinsic cause of the strain-rate effects of brittle granular materials. Using the discrete element method (OEM), the energy distributions in the brittle granular material under confined compression are discussed. It is observed that the input work is mainly transformed into the frictional dissipation, and the frictional dissipation under dynamic loading is higher than that under quasi-static loading corresponding to the same breakage extent. The reason is that more fragmentation debris is produced during dynamic breakage of single grains, which promotes particle rearrangement and the corresponding frictional dissipation significantly.