The nonlinear behavior of a rigid rotor supported by herringbone grooved journal gas bearings (HGJBs) was investigated in this study. The two-dimensional narrow groove theory (2D-NGT) was adopted to model the HGJBs. A set of integrated rotor-bearing state equations were built by coupling the rotor motion equations and the bearing Reynolds equation. An implicit integrator with adaptive time step method was used to solve those state equations continuously. Two low-stability HGJBs were implemented to experimentally demonstrate and analyze the appearance of self-excitation motions. The theoretical model was successfully validated by the experimental data on predicting the onset speed of the sub-synchronous vibration of the HGJB-rotor system and the whirl frequency ratio. The predicted limit cycle amplitude increases as the speed increases until the rotor contacts with the bearing surface, which leads to a bearing failure. Forward conical mode dominates the self-excited motion during the whole speed range of self-excited motion. The prediction shows that the HGJB-rotor system can still operate in a stable, even though the rotor is installed vertically, i.e., without static load on the bearings. This is a distinct advantage in comparison to plain bearings. As the static load applies on the bearings increases, the onset speed of sub-synchronous vibration increases as well. For the investigated rotor-bearing system, an increase of the onset speed of sub-synchronous vibration from 36 krpm to 75 krpm is predicted as the static load increases from 0 to 4 times of the rotor weight. This indicates an increased HGJB stability with increased static load. The rotor orbits show complex shapes when the imbalance excitation is considered in the simulation. Both synchronous frequency and whirl frequency are shown in the spectral analysis. Moreover, the speed range of self-excited motion reduces from [38,48] krpm to [40,42] krpm as the imbalance increases from 0 to 40 mgmm.