Higher risk for long-term cognitive and behavioral impairments is one of the hallmarks of extreme prematurity (EP) and pregnancy-associated fetal adverse conditions such as intrauterine growth restriction (IUGR). While neurodevelopmental delay and abnormal brain function occur in the absence of overt brain lesions, these conditions have been recently associated with changes in microstructural brain development. Recent imaging studies indicate changes in brain connectivity, in particular involving the white matter fibers belonging to the cortico-basal ganglia-thalamic loop. Furthermore, EP and IUGR have been related to altered brain network architecture in childhood, with reduced network global capacity, global efficiency and average nodal strength. In this study, we used a connectome analysis to characterize the structural brain networks of these children, with a special focus on their topological organization. On one hand, we confirm the reduced averaged network node degree and strength due to EP and IUGR. On the other, the decomposition of the brain networks in an optimal set of clusters remained substantially different among groups, talking in favor of a different network community structure. However, and despite the different community structure, the brain networks of these high-risk school-age children maintained the typical small-world, rich-club and modularity characteristics in all cases. Thus, our results suggest that brain reorganizes after EP and IUGR, prioritizing a tight modular structure, to maintain the small-world, rich-club and modularity characteristics. By themselves, both extreme prematurity and IUGR bear a similar risk for neurocognitive and behavioral impairment, and the here defined modular network alterations confirm similar structural changes both by IUGR and EP at school age compared to control. Interestingly, the combination of both conditions (IUGR + EP) does not result in a worse outcome. In such cases, the alteration in network topology appears mainly driven by the effect of extreme prematurity, suggesting that these brain network alterations present at school age have their origin in a common critical period, both for intrauterine and extrauterine adverse conditions.