The dimerized quantum magnet BaCuSi2O6 was proposed as an example of "dimensional reduction" arising near the magnetic-field-induced quantum critical point (QCP) due to perfect geometrical frustration of its interbilayer interactions. We demonstrate by high-resolution neutron spectroscopy experiments that the effective intrabilayer interactions are ferromagnetic, thereby excluding frustration. We explain the apparent dimensional reduction by establishing the presence of three magnetically inequivalent bilayers, with ratios 3:2:1, whose differing interaction parameters create an extra field-temperature scaling regime near the QCP with a nontrivial but nonuniversal exponent. We demonstrate by detailed quantum Monte Carlo simulations that the magnetic interaction parameters we deduce can account for all the measured properties of BaCuSi2O6, opening the way to a quantitative understanding of nonuniversal scaling in any modulated layered system.