We propose a non-contact surface finishing method for brittle substrates by ion beam etching and we experimentally demonstrate polishing of (100) single crystal diamond surface. We model and simulate the polishing process, and verify the results experimentally by monitoring individual defects during the surface treatment. Rapid flattening of scratches and digs, as typically present on brittle substrates after mechanical polishing, is observed: trench depth is typically removed by 95% in less than 30 min. The polishing method relies on physical bombardment of the substrate surface with accelerated inert gas ions, rendering it highly versatile and applicable to a wide variety of materials.