We present accurate time delays for the quadruply imaged quasar HE 0435-1223. The delays were measured from 575 independent photometric points obtained in the R-band between January 2004 and March 2010. With seven years of data, we clearly show that quasar image A is affected by strong microlensing variations and that the time delays are best expressed relative to quasar image B. We measured Delta t(BC) = 7.8 +/- 0.8 days, Delta t(BD) = -6.5 +/- 0.7 days and Delta t(CD) = -14.3 +/- 0.8 days. We spacially deconvolved HST NICMOS2 F160W images to derive accurate astrometry of the quasar images and to infer the light profile of the lensing galaxy. We combined these images with a stellar population fitting of a deep VLT spectrum of the lensing galaxy to estimate the baryonic fraction, f(b), in the Einstein radius. We measured f(b) = 0.65(-0.10)(+0.13) if the lensing galaxy has a Salpeter IMF and f(b) = 0.45(-0.07)(+0.04) if it has a Kroupa IMF. The spectrum also allowed us to estimate the velocity dispersion of the lensing galaxy, sigma(ap) = 222 +/- 34 km s(-1). We used f(b) and sigma(ap) to constrain an analytical model of the lensing galaxy composed of an Hernquist plus generalized NFW profile. We solved the Jeans equations numerically for the model and explored the parameter space under the additional requirement that the model must predict the correct astrometry for the quasar images. Given the current error bars on f(b) and sigma(ap), we did not constrain H-0 yet with high accuracy, i.e., we found a broad range of models with chi(2) < 1. However, narrowing this range is possible, provided a better velocity dispersion measurement becomes available. In addition, increasing the depth of the current HST imaging data of HE 0435-1223 will allow us to combine our constraints with lens reconstruction techniques that make use of the full Einstein ring that is visible in this object.