The unavailability of highly transparent materials in the mid-infrared has been the main limitation in the development of ultra-sensitive molecular sensors or cavity-based spectroscopy applications. Whispering gallery mode microresonators have attained ultra-high-quality (Q) factor resonances in the near-infrared and visible. Here we report ultra-high Q factors in the mid-infrared using polished alkaline earth metal fluoride crystals. Using an uncoated chalcogenide tapered fibre as a high-ideality coupler in the mid-infrared, we study via cavity ringdown technique the losses of BaF2, CaF2, MgF2 and SrF2 microresonators. We show that MgF2 is limited by multiphonon absorption by studying the temperature dependence of the Q factor. In contrast, in SrF2 and BaF2 the lower multiphonon absorption leads to ultra-high Q factors at 4.5 mu m. These values correspond to an optical finesse of F > 1 x 10(5), the highest value achieved for any type of mid-infrared resonator to date.