The formation of temporal dissipative solitons in optical microresonators enables compact, high-repetition rate sources of ultrashort pulses as well as low noise, broadband optical frequency combs with smooth spectral envelopes. Here we study the influence of the microresonator mode spectrum on temporal soliton formation in a crystalline MgF2 microresonator. While an overall anomalous group velocity dispersion is required, it is found that higher order dispersion can be tolerated as long as it does not dominate the resonator's mode structure. Avoided mode crossings induced by linear mode coupling in the resonator mode spectrum are found to prevent soliton formation when affecting resonator modes close to the pump laser frequency. The experimental observations are in excellent agreement with numerical simulations based on the nonlinear coupled mode equations. The presented results provide for the first time design criteria for the generation of temporal solitons in optical microresonators.