High-efficiency amorphous silicon (a-Si:H) solar cells were deposited with different thicknesses of the p-type amorphous silicon carbide layer on substrates of varying roughness. We observed a light-induced open-circuit voltage (Voc) increase upon light soaking for thin p-layers, but a decrease for thick p-layers. Further, the Voc increase is enhanced with increasing substrate roughness. After correction of the p-layer thickness for the increased surface area of rough substrates, we can exclude varying the effective p-layer thickness as the cause of the substrate roughness dependence. Instead, we explain the observations by an increase of the dangling-bond density in both the p-layer—causing a Voc increase—and in the intrinsic absorber layer, causing a Voc decrease. We present a mechanism for the light-induced increase and decrease, justified by the investigation of light-induced changes of the p-layer and supported by Advanced Semiconductor Analysis simulation. We conclude that a shift of the electron quasi-Fermi level towards the conduction band is the reason for the observed Voc enhancements, and poor amorphous silicon quality on rough substrates enhances this effect.