Recently developed self-consistent models for beam-wave interaction simulations of a gyrotron oscillator are used to study spurious backward-wave instabilities in smooth-wall metallic beam ducts. Starting from a simple case with a basic geometry, the influence of the beam duct geometry and the magnetic field profile on the instabilities threshold and on the quality of the electron beam are studied.