Neutron diffraction and nuclear quadrupole resonance (NQR) measurements were employed to investigate magnetic order in the nonferroelectric phase preceding the low-temperature multiferroic state in FeTe2O5Br. Refinement of the neutron diffraction data and simulations of Br-79,Br-81 NQR spectra reveal that the incommensurate magnetic ordering in the nonferroelectric state comprises amplitude-modulated magnetic moments, similar to that seen in the multiferroic state. The two ordered states differ in the orientation of the magnetic moments and phase shifts between modulation waves. Surprisingly, all symmetry restrictions for the electric polarization are absent in both states. The different ferroelectric responses of the two states are thus argued to arise from the differences in the phase shifts between certain modulation waves, which cancel out in the nonferrolectric state. DOI: 10.1103/PhysRevB.87.144408