Parasitic oscillations excitation is one of the main point hindering high-power gyrotron operation for fusion application. These instabilities, besides being dangerous for the gyrotron components, could possibly degrade
the electron beam quality before it enters the cavity. In order to study the instabilities that may occur in a realistic beam duct upstream of the gyrotron cavity, the self-consistent linear and spectral code TWANGlinspec has been
modified. The large inhomogeneities in the smooth-wall beam duct geometry or in the magnetic field profile required the implementation of a numerical approach using a hybrid finite element method. The new model allows to
characterize a large number of potentially spurious transverse electric (TE) modes.

Compared to previous studies on gyrotron beam duct instabilities, an extended interaction space including also the gyrotron cavity has been considered. The role of the connecting part called spacer between the beam duct and the cavity is highlighted and it is shown that the gyro backward-wave TE modes excited in this region generally have their minimum starting current. The backward-wave nature of the parasitic oscillations is responsible to their strong electron
velocity spread dependency, as shown with the new code TWANGlinspread. Nonlinear models were also used and allowed to evaluate the effect of the parasitic oscillations on the electron beam quality. These
parasitic oscillations induce a large electron beam energy spread and subsequently a significant reduction of the gyrotron efficiency. However, the competition between the parasitic and the operating mode could play an
important role and a multimode model should be used to fully study the situation in one single simulation.

For the first time, a self-consistent electron beam wave interaction is simulated in the presence of a lossy dielectric layer in the smooth-wall beam duct. In TWANGlinspec, the transverse structure of the TE mode is adapted to
the solution of the complex cold dispersion relation of an infinite homogeneous dielectric coated cylindrical waveguide. Before considering the realistic situation, the dispersion relation formulation had to be adapted for
dielectric materials with very high losses and the validity of the TE pure mode ($E_z = 0$) assumed in TWANGlinspec had to be assessed for SiC or BeOSiC materials. The effect of the dielectric layer on the parasitic starting
current is large for parasitic oscillations localized at the end of the beam duct and in the spacer region.

For a real case, the dual frequency gyrotron for the Tokamak à Configuration Variable (TCV), but with a smooth-wall SiC dielectric layer and with a realistic electron beam including velocity spread, the parasitic oscillations starting current is increased to a level higher than the operating beam current. The excitation of these parasitic oscillations is thus not expected to occur.

The new code TWANGlinspec, convenient for starting current calculations, has also been applied to high-power gyrotron start-up studies. These gyrotrons often suffer from the excitation of competing modes excited in the cavity
during the start-up phase, when the electron beam parameters are varying to reach their nominal values. For a realistic start-up situation for the TCV dual-frequency gyrotron, the simulations with TWANGlinspec are in a remarkable
agreement with the experiments, additionally validating the code TWANGlinspec.