The revised NEO Personality Inventory (NEOPI-R), popularly known as the five-factor model, defines five personality factors: Neuroticism, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness. The structural correlates of these personality factors are still a matter of debate. In this work, we examine the impact of subtle cognitive deficits on structural substrates of personality in the elderly using DTI derived white matter (WM) integrity measure, Fractional Anisotropy (FA). We employed canonical correlation analysis (CCA) to study the relationship between personality factors of the NEOPI-R and FA measures in two population groups: healthy controls and MCI. Agreeableness was the only personality factor to be associated with FA patterns in both groups. Openness was significantly related to FA data in the MCI group and the inverse was true for Conscientiousness. Furthermore, we generated saliency maps using bootstrapping strategy which revealed a larger number of positive correlations in healthy aging in contrast to the MCI status. The MCI group was found to be associated with a predominance of negative correlations indicating that higher Agreeableness and Openness scores were mostly related to lower FA values in interhemispheric and cortico-spinal tracts and a limited number of higher FA values in cortico-cortical and cortico-subcortical connection. Altogether these findings support the idea that WM microstructure may represent a valid correlate of personality dimensions and also indicate that the presence of early cognitive deficits led to substantial changes in the associations between WM integrity and personality factors.