Large excitatory synapses are found at specific points in the neuronal circuits of the auditory brainstem, to enable fast information transfer and the preservation of acoustic timing information. The extracellular cues and signaling mechanisms that lead to the development of these specialized synaptic connections, exemplified by the calyx of Held in the medial nucleus of the trapezoid body (MNTB), are still largely unknown. Here, we investigate the role of BMP signaling for the early development of the ventral cochlear nucleus (VCN) and MNTB, and for the initial formation of the calyx of Held synaptic connection. We used conditional alleles of two BMP type-1 receptors in the background of a constitutive BMPR1b knock-out (KO), or else a conditional allele of SMAD4. The conditional alleles were recombined by the Krox20(Cre) mouse line that is active around mid-gestation in rhombomeres (r) 3 and 5 from which the VCN and MNTB are derived; alternatively, virus-mediated Cre-expression was performed early postnatally in the VCN. The data show that embryonic SMAD-dependent BMP-signaling in r3 and r5 contributes to the histogenesis of auditory brainstem nuclei. On the other hand, BMP-receptor signaling early postnatally in presynaptic neurons of the calyx of Held projection is necessary for correct axon branch retraction, which suggests a cell-autonomous role of presynaptic BMP-receptors in synapse elimination at the developing calyx of Held. Thus, our work dissects developmentally early and late roles of BMP-signaling for the formation of auditory brainstem nuclei, and the highly specialized synaptic connectivity in these structures.