Face recognition is of major social importance and involves highly selective brain regions thought to be organized in a distributed functional network. However, the exact architecture of interconnections between these regions remains unknown. We used functional magnetic resonance imaging to identify face-responsive regions in 22 participants and then employed diffusion tensor imaging with probabilistic tractography to establish the white-matter pathways between these functionally defined regions. We identified strong white-matter connections between the occipital face area (OFA) and fusiform face area (FFA), with a significant right-hemisphere predominance. We found no evidence for direct anatomical connections between FFA and superior temporal sulcus (STS) or between OFA and STS, contrary to predictions based on current cognitive models. Instead, our findings point to segregated processing along a ventral extrastriate visual pathway to OFA-FFA and another more dorsal system connected to STS and frontoparietal areas. In addition, early occipital areas were found to have direct connections to the amygdala, which might underlie a rapid recruitment of limbic brain areas by visual inputs bypassing more elaborate extrastriate cortical processing. These results unveil the structural neural architecture of the human face recognition system and provide new insights on how distributed face-responsive areas may work together.