As organic solar cells (OSCs) and perovskite solar cells (PVSCs) move closer to commercialization, further efforts toward optimizing both cell efficiency and stability are needed. As interfaces strongly affect device performance and degradation processes, interfacial engineering by employing various materials as hole transport layers (HTLs) and electron transport layers (ETLs) has been a very active field of research in OSCs and PVSCs. Among them, inorganic materials exhibit significant advantages in promoting device performance due to their excellent charge transporting properties and intrinsic thermal and chemical robustness. In this review, an extensive overview is provided of inorganic semiconductors such as copper-based ones with emphasis on copper iodide and copper thiocyanate, transition metal chalcogenides, nitrides and carbides as well as hybrid materials based on these inorganic compounds that have been recently employed as HTLs and ETLs in OSCs and PVSCs. Following a short discussion of the main optoelectronic and physical properties that interfacial materials used as HTLs and ETLs should possess, the functionalities of the aforementioned materials as interfacial, charge transport, layers in OSCs and PVSCs are discussed in depth. It is concluded by providing guidelines for further developments that could significantly extend the implementation of these materials in solar cells.