Through first-principles simulation methods, we assign the origin of Fermi-level pinning at GaAs surfaces and interfaces to the bistability between the As-As dimer and two As dangling bonds, which transform into each other upon charge trapping. This defect is shown to be naturally formed both at GaAs surfaces upon oxygen deposition and in the near-interface substoichiometric oxide. Using electron-counting arguments, we infer that the identified defect occurs in opposite charge states. The Fermi-level pinning then results from the amphoteric nature of this defect which drives the Fermi level to its defect level. These results account for the experimental characterization at both GaAs surfaces and interfaces within a unified picture, wherein the role of As antisites is elucidated.