Normal and pathologic neurobiological processes influence brain morphology in coordi-nated ways that give rise to patterns of structural covariance (PSC) across brain regions and individuals during brain aging and diseases. The genetic underpinnings of these pat-terns remain largely unknown. We apply a stochastic multivariate factorization method to a diverse population of 50,699 individuals (12 studies and 130 sites) and derive data- driven, multi -scale PSCs of regional brain size. PSCs were significantly correlated with 915 genomic loci in the discovery set, 617 of which are newly identified, and 72% were independently replicated. Key pathways influencing PSCs involve reelin signaling, apoptosis, neurogenesis, and appendage development, while pathways of breast cancer indicate potential interplays between brain metastasis and PSCs associated with neuro-degeneration and dementia. Using support vector machines, multi -scale PSCs effectively derive imaging signatures of several brain diseases. Our results elucidate genetic and bio-logical underpinnings that influence structural covariance patterns in the human brain.