White adipose tissue (WAT) is a cellularly heterogeneous endocrine organ that not only serves as an energy reservoir, but also actively participates in metabolic homeostasis. Among the main constituents of adipose tissue are adipocytes, which arise from adipose stem and progenitor cells (ASPCs). While it is well known that these ASPCs reside in the stromal vascular fraction (SVF) of adipose tissue, their molecular heterogeneity and functional diversity is still poorly understood. Driven by the resolving power of single-cell transcriptomics, several recent studies provided new insights into the cellular complexity of ASPCs among different mammalian fat depots. In this review, we present current knowledge on ASPCs, their population structure, hierarchy, fat depot-specific nature, function, and regulatory mechanisms, and discuss not only the similarities, but also the differences between mouse and human ASPC biology.