We consider a class of singular perturbations to the stochastic heat equation or semilinear variations thereof. The interesting feature of these perturbations is that, as the small parameter epsilon tends to zero, their solutions converge to the 'wrong' limit, i.e. they do not converge to the solution obtained by simply setting epsilon = 0. A similar effect is also observed for some (formally) small stochastic perturbations of a deterministic semilinear parabolic PDE. Our proofs are based on a detailed analysis of the spatially rough component of the equations, combined with a judicious use of Gaussian concentration inequalities.