Background: The increasingly common applications of machine-learning schemes to atomic-scale simulations have triggered efforts to better understand the mathematical properties of the mapping between the Cartesian coordinates of the atoms and the variety of representations that can be used to convert them into a finite set of symmetric descriptors or features. Methods: Here, we analyze the sensitivity of the mapping to atomic displacements, using a singular value decomposition of the Jacobian of the transformation to quantify the sensitivity for different configurations, choice of representations and implementation details. Results: We show that the combination of symmetry and smoothness leads to mappings that have singular points at which the Jacobian has one or more null singular values (besides those corresponding to infinitesimal translations and rotations). This is in fact desirable, because it enforces physical symmetry constraints on the values predicted by regression models constructed using such representations. However, besides these symmetry-induced singularities, there are also spurious singular points, that we find to be linked to the incompleteness of the mapping, i.e. the fact that, for certain classes of representations, structurally distinct configurations are not guaranteed to be mapped onto different feature vectors. Additional singularities can be introduced by a too aggressive truncation of the infinite basis set that is used to discretize the representations. Conclusions: These results exemplify the subtle issues that arise when constructing symmetric representations of atomic structures, and provide conceptual and numerical tools to identify and investigate them in both benchmark and realistic applications.