In this paper we provide the characterization of all finite-dimensional Heath-Jarrow-Morton models that admit arbitrary initial yield curves. It is well known that affine term structure models with time-dependent coef- ficients (such as the Hull-White extension of the Vasicek short rate model) perfectly fit any initial term structure. We find that such affine models are in fact the only finite-factor term structure models with this property. We also show that there is usually an invariant singular set of initial yield curves where the affine term structure model becomes time-homogeneous. We also argue that other than functional dependent volatility structures - such as local state dependent volatility structures - cannot lead to finite-dimensional realizations. Finally, our geometric point of view is illustrated by several examples.