Gyrotrons belong to the family of high-power coherent radiation sources known as Electron Cyclotron Masers (ECMs) and are based on the physical mechanism of the ECM-instability, converting electron rotational kinetic energy into coherent electromagnetic radiation. The worldwide gyrotron R&D is mainly driven by the application in heating a magnetically confined fusion plasma, which requires coherent radiation sources with MW power-level in the sub-THz frequency range. In the last two decades, an application for gyrotrons emerged in the field of Nuclear Magnetic Resonance (NMR) spectroscopy, where a dramatic enhancement in sensitivity can be achieved via Dynamic Nuclear Polarization (DNP), requiring a low-power (~1-10W), sub-THz frequency (<200GHz) coherent radiation. The subject of this thesis is a gyrotron prototype developed at SPC/EPFL for the DNP-application. It is designed for continuous mode (CW)-operation on the TE7,2-mode and has a maximum radio-frequency (RF)-power of P=150W at a frequency of f=260.5GHz. The DNP-gyrotron has demonstrated to be an ideal test-bench for fundamental research on the beam-wave interaction process. The weakly overmoded gyrotron cavity is such that transverse mode competition can be neglected and the studied dynamical regimes concentrate on the 1D-longitudinal dynamics. The main topic of this work concentrates on the experimental measurements and numerical modeling of novel non-stationary regimes, characterized by a multi-frequency spectrum and a modulated RF-power. These experimental results have shown that the very fast dynamics (nanosecond time-scale) observed in non-stationary regimes is such that the usual assumption, in which the cavity electromagnetic field is not varying during the electron time of flight, is no more valid. To overcome this assumption a new model based on a Particle-In-Cell (PIC) approach has been developed and a new code TWANG-PIC has been written and successfully exploited. Also, the linear regime has been revisited from the theoretical point of view by developing a new moment-based model. Based on this model a new code TWANGLIN has been written and used for a detailed analysis of the experimentally measured threshold conditions (starting current) covering operating points from forward to backward-wave gyrotron regimes. Among a large variety of non-stationary regimes, that is described and analyzed, a novel specific nanosecond-pulsed regime was studied, in which the multi-frequency spectrum consists of frequency-equidistant, phase-locked sidebands. This novel regime may open up new applications for gyrotrons. For the first time it has also been possible to experimentally investigate, and model via numerical simulations, the dynamical properties from the linear regime up to chaotic regimes. The numerical simulations with TWANG-PIC are in good qualitative agreement with the experimental results and showed that the observed non-stationary regimes are associated to non-linear axial mode-competition. Another important task was to configure the gyrotron for the DNP-NMR spectroscopy application. Several state-of-the-art features have been included, such as a continuous frequency-tuning over 1.2GHz by varying several control parameters simultaneously, a fast (<15kHz) frequency-modulation over ~100MHz and a feedback-controller for stabilizing RF-parameters. Currently the gyrotron is routinely and successfully operated on a 400MHz DNP-NMR experiment.