Both numerical simulations and data-driven methods have been applied in dam's displacement modeling. For monitored displacement data-driven methods, the physical mechanism and structural correlations were rarely discussed. In order to take the spatial and temporal correlations among all monitoring points into account, we took the first step toward integrating the finite element method into a data-driven model. As the data-driven method, we selected the random coefficient model, which can make each explanatory variable coefficient of all monitoring points following one or several normal distributions. In this way, explanatory variables are constrained. Another contribution of the proposed model is that the actual elastic modulus at each monitoring point can be back-calculated. Moreover, with a Lagrange polynomial interpolation, we can obtain the distribution field of elastic modulus, rather than gaining one value for the whole dam in previous studies. The proposed model was validated by a case study of the concrete arch dam in Jinping-I hydropower station. It has a better prediction precision than the random coefficient model without the finite element method.