The microscopic origin of the intermediate phase in two prototypical covalently bonded A(x)B(1-x) network glass forming systems, where A=Ge or Si, B=Se, and 0 <= x <= 1, was investigated by combining neutron diffraction with first-principles molecular-dynamics methods. Specifically, the structure of glassy GeSe4 and SiSe4 was examined, and the calculated total structure factor and total pair-correlation function for both materials are in good agreement with experiment. The structure of both glasses differs markedly from a simple model comprising undefective AB(4) corner-sharing tetrahedra in which all A atoms are linked by B-2 dimers. Instead, edge-sharing tetrahedra occur and the twofold coordinated Se atoms form three distinct structural motifs, namely, Se-Se-2, Se-SeGe (or Se-SeSi), and Se-Ge-2 (or Se-Si-2). This identifies several of the conformations that are responsible for the structural variability in GexSe1-x and SixSe1-x glasses, a quantity that is linked to the finite width of the intermediate phase window.