We carry out first-principles calculations of band gaps of cubic inorganic perovskites belonging to the class CsBX3, with B = Pb, Sn and X = Cl, Br, I. We use the quasi particle self-consistent GW method with efficient vertex corrections to calculate the electronic structure of the studied materials. We demonstrate the importance of including the higher lying core and semicore shells among the valence states. For a meaningful comparison with experimental values, we account for thermal vibrations and disorder through ab initio molecular dynamics. Additionally, we calculate the spin orbit coupling at levels of theory of increasing accuracy and show that semilocal density functionals significantly underestimate these corrections. We show that all of these effects need to be properly included in order to obtain reliable predictions for the band gaps of halide perovskites.