We study correlation functions involving generalized ANEC operators of the form integral dx-x-n+2T--x -> in four dimensions. We compute two, three, and four-point functions involving external scalar states in both free and holographic Conformal Field Theories. From this information, we extract the algebra of these light-ray operators. We find a global subalgebra spanned by n = {-2, -1, 0, 1, 2} which annihilate the conformally invariant vacuum and transform among themselves under the action of the collinear conformal group that preserves the light-ray. Operators outside this range give rise to an infinite central term, in agreement with previous suggestions in the literature. In free theories, even some of the operators inside the global subalgebra fail to commute when placed at spacelike separation on the same null-plane. This lack of commutativity is not integrable, presenting an obstruction to the construction of a well defined light-ray algebra at coincident x-> coordinates. For holographic CFTs the behavior worsens and operators with n not equal -2 fail to commute at spacelike separation. We reproduce this result in the bulk of AdS where we present new exact shockwave solutions dual to the insertions of these (exponentiated) operators on the boundary.