Charge carrier collection in silicon heterojunction solar cells occurs via intrinsic/doped hydrogenated amorphous silicon layer stacks deposited on the crystalline silicon wafer surfaces. Usually, both the electron and hole collecting stacks are externally capped by an n-type transparent conductive oxide, which is primarily needed for carrier extraction. Earlier, it has been demonstrated that the mere presence of such oxides can affect the carrier recombination in the crystalline silicon absorber. Here, we present a detailed investigation of the impact of this phenomenon on both the electron and hole collecting sides, including its consequences for the operating voltages of silicon heterojunction solar cells. Based on our findings, we define guiding principles for improved passivating contact design for high-efficiency silicon solar cells.