Galaxies located in the environment or along the line of sight towards gravitational lenses can significantly affect lensing observables, and can lead to systematic errors on the measurement of H-0 from the time-delay technique. We present the results of a systematic spectroscopic identification of the galaxies in the field of view of the lensed quasar HE 0435-1223 using the W. M. Keck, Gemini and ESO-Very Large telescopes. Our new catalogue triples the number of known galaxy redshifts in the direct vicinity of the lens, expanding to 102 the number of measured redshifts for galaxies separated by less than 3 arcmin from the lens. We complement our catalogue with literature data to gather redshifts up to 15 arcmin from the lens, and search for galaxy groups or clusters projected towards HE 0435-1223. We confirm that the lens is a member of a small group that includes at least 12 galaxies, and find 8 other group candidates near the line of sight of the lens. The flexion shift, namely the shift of lensed images produced by high-order perturbation of the lens potential, is calculated for each galaxy/group and used to identify which objects produce the largest perturbation of the lens potential. This analysis demonstrates that (i) at most three of the five brightest galaxies projected within 12 arcsec of the lens need to be explicitly used in the lens models, and (ii) the groups can be treated in the lens model as an external tidal field (shear) contribution.