A systematic study of the factorization of long-range azimuthal two-particle correlations into a product of single-particle anisotropies is presented as a function of pT and η of both particles and as a function of the particle multiplicity in PbPb and pPb collisions. The data were taken with the CMS detector for PbPb collisions at sNN=2.76 TeV and pPb collisions at sNN=5.02 TeV, covering a very wide range of multiplicity. Factorization is observed to be broken as a function of both particle pT and η. When measured with particles of different pT, the magnitude of the factorization breakdown for the second Fourier harmonic reaches 20% for very central PbPb collisions but decreases rapidly as the multiplicity decreases. The data are consistent with viscous hydrodynamic predictions, which suggest that the effect of factorization breaking is mainly sensitive to the initial-state conditions rather than to the transport properties (e.g., shear viscosity) of the medium. The factorization breakdown is also computed with particles of different η. The effect is found to be weakest for mid-central PbPb events but becomes larger for more central or peripheral PbPb collisions, and also for very-high-multiplicity pPb collisions. The η-dependent factorization data provide new insights to the longitudinal evolution of the medium formed in heavy ion collisions.