Electron beam instabilities occurring in a gyrotron electron beam can induce an energy spread which might significantly deteriorate the gyrotron efficiency. Three types of instabilities are considered to explain the important discrepancy found between the theoretical and experimental efficiency in the case of quasi-optical gyrotrons (QOG): the electron cyclotron maser instability (ECMI), the electrostatic Bernstein instability (BI) and the Langmuir instability (LI). When the magnetic field gradient in drift tubes of QOG is low, the ECMI can develop in the drift tube at very low electron beam currents. Experimental measurements show that with a proper choice of absorbing structures in the beam tunnel, this instability can be suppressed. At high beam currents, the BI can induce a significant energy spread at the entrance of the interaction region. The induced energy spread scales approximately linearly with the electron beam density and for QOG one observes that the beam density is significantly higher than the beam density of an equivalent cylindrical cavity gyrotron. (C) 1998 American Institute of Physics. [S1070-664X(98)01906-5]