The phase of a quantum state may not return to its original value after the system's parameters cycle around a closed path; instead, the wave function may acquire a measurable phase difference called the Berry phase. Berry phases typically have been accessed through interference experiments. Here, we demonstrate an unusual Berry phase-induced spectroscopic feature: a sudden and large increase in the energy of angular-momentum states in circular graphene p-n junction resonators when a relatively small critical magnetic field is reached. This behavior results from turning on a p Berry phase associated with the topological properties of Dirac fermions in graphene. The Berry phase can be switched on and off with small magnetic field changes on the order of 10 millitesla, potentially enabling a variety of optoelectronic graphene device applications.