We exhibit non-equivariant perturbations of the blowup solutions constructed in [18] for energy critical wave maps into $\mathbb{S}^2$. Our admissible class of perturbations is an open set in some sufficiently smooth topology and vanishes near the light cone. We show that the blowup solutions from [18] are rigid under such perturbations, including the space-time location of blowup. As blowup is approached, the dynamics agrees with the classification obtained in [7], and all six symmetry parameters converge to limiting values. Compared to the previous work [16] in which the rigidity of the blowup solutions from [18] under equivariant perturbations was proved, the class of perturbations considered in the present work does not impose any symmetry restrictions. Separation of variables and decomposing into angular Fourier modes leads to an infinite system of coupled nonlinear equations, which we solve for small admissible data. The nonlinear analysis is based on the distorted Fourier transform, associated with an infinite family of Bessel type Schrödinger operators on the half-line indexed by the angular momentum $n$. A semi-classical WKB-type spectral analysis relative the parameter $\hbar=\frac{1}{n+1}$ for large $|n|$ allows us to effectively determine the distorted Fourier basis for the entire infinite family. Our linear analysis is based on the global Liouville-Green transform as in the earlier work [4, 5].