In many problems such as phase retrieval, molecular biology, source localization, and sensor array calibration, one can measure vector differences between pairs of points and attempt to recover the position of these points; this class of problems is called vector geometry problems (VGPs). A closely related field studies distance geometry problems (DGPs), where only the Euclidean distance between pairs of points is available. This has been extensively studied in the literature and is often associated with Euclidean distance matrices (EDMs). Although similar to DGPs, VGPs have received little attention in the literature; our goal is to fill in this gap and introduce a framework to solve VGPs. Inspired by EDM-related approaches, we arrange the differences in what we call a coordinate difference matrix (CDM) and introduce a methodology to reconstruct a set of points from CDM entries. We first propose a reconstruction scheme in 1D and then generalize it to higher dimensions. We show that our algorithm is optimal in the least-squares sense, even when we have only access to partial measurements. In addition, we provide necessary and sufficient conditions on the number and structure of measurements needed for a successful reconstruction, as well as a comparison with EDMs. In particular we show that compared to EDMs, CDMs are simpler objects, both from an algorithmic and a theoretical point of view. Therefore, CDMs should be favored over EDMs whenever vector differences are available. In the presence of noise, we provide a statistical analysis of the reconstruction error. Finally, we apply the established knowledge to five practical problems to demonstrate the versatility of this theory and showcase the wide range of applications covered by the CDM framework.