We show that the TE-TM modes splitting and the structure anisotropy of a semiconductor microcavity combine into a non-Abelian gauge field for exciton-polaritons or cavity photons. The field texture can be tuned simply by rotating the sample and ranges continuously from a Rashba to a monopolar field. In the noninteracting regime, the latter leads to remarkable focusing and conical diffraction effects. In the interacting regime, the spin-orbit coupling induces a breakdown of superfluidity. The spatially homogeneous flows become unstable and dynamically evolve into spin textured states, such as stripes or domain walls.