Patients with Temporal Lobe Epilepsy (TLE) suffer from widespread subtle white matter abnormalities and abnormal functional connectivity extending beyond the affected lobe, as revealed by Diffusion Tensor MR Imaging, volumetric and functional MRI studies. Diffusion Spectrum Imaging (DSI) is a diffusion imaging technique with high angular resolution for improving the mapping of white matter pathways. In this study, we used DSI, connectivity matrices and topological measures to investigate how the alteration in structural connectivity influences whole brain structural networks. Eleven patients with right-sided TLE and hippocampal sclerosis and 18 controls underwent our DSI protocol at 3T. The cortical and subcortical grey matters were parcellated into 86 regions of interest and the connectivity between every region pair was estimated using global tractography and a connectivity matrix (the adjacency matrix of the structural network). We then compared the networks of patients and controls using topological measures. In patients, we found a higher characteristic path length and a lower clustering coefficient compared to controls. Local measures at node level of the clustering and efficiency showed a significant difference after a multiple comparison correction (Bonferroni). These significant nodes were located within as well outside the temporal lobe, and the localisation of most of them was consistent with regions known to be part of epileptic networks in TLE. Our results show altered connectivity patterns that are concordant with the mapping of functional epileptic networks in patients with TLE. Further studies are needed to establish the relevance of these findings for the propagation of epileptic activity, cognitive deficits in medial TLE and outcome of epilepsy surgery in individual patients.