Given a possibly reducible and non-reduced spectral cover pi: X -> C over a smooth projective complex curve C we determine the group of connected components of the Prym variety Prym(X/C). As an immediate application we show that the finite group of n-torsion points of the Jacobian of C acts trivially on the cohomology of the twisted SLn-Higgs moduli space up to the degree which is predicted by topological mirror symmetry. In particular this yields a new proof of a result of Harder-Narasimhan, showing that this finite group acts trivially on the cohomology of the twisted SLn stable bundle moduli space.