We show in experiments that a long, underdense, relativistic proton bunch propagating in plasma undergoes the oblique instability, which we observe as filamentation. We determine a threshold value for the ratio between the bunch transverse size and plasma skin depth for the instability to occur. At the threshold, the outcome of the experiment alternates between filamentation and self-modulation instability (evidenced by longitudinal modulation into microbunches). Time-resolved images of the bunch density distribution reveal that filamentation grows to an observable level late along the bunch, confirming the spatiotemporal nature of the instability. We provide a rough estimate of the amplitude of the magnetic field generated in the plasma by the instability and show that the associated magnetic energy increases with plasma density.