Stability of a dense plasma confined by a rotating magnetic field