The ability of subfilter-scale (SFS) models to reproduce the statistical properties of SFS stresses and energy transfers over heterogeneous surface roughness is key to improving the accuracy of large-eddy simulations of the atmospheric boundary layer. In this study, several SFS models are evaluated a priori using experimental data acquired downwind of a rough-to-smooth transition in a wind tunnel. The SFS models studied include the eddy-viscosity, similarity, non-linear and a mixed model consisting of a combination of the eddy-viscosity and non-linear models. The dynamic eddy-viscosity model is also evaluated. The experimental data consist of vertical and horizontal planes of high-spatial-resolution velocity fields measured using particle image velocimetry. These velocity fields are spatially filtered and used to calculate SFS stresses and SFS transfer rates of resolved kinetic energy. Coefficients for each SFS model are calculated by matching the measured and modelled SFS energy transfer rates. For the eddy-viscosity model, the Smagorinsky coefficient is also evaluated using a dynamic procedure. The model coefficients are found to be scale dependent when the filter scales are larger than the vertical measurement height and fall into the production subrange of the turbulence where the flow scales are anisotropic. Near the surface, the Smagorinsky coefficient is also found to decrease with distance downwind from the transition, in response to the increase in mean shear as the flow adjusts to the smooth surface. In a priori tests, the ability of each model to reproduce statistical properties of the SFS stress is assessed. While the eddy-viscosity model has low spatial correlation with the measured stress, it predicts mean stresses with the same accuracy as the other models. However, the deficiency of the eddy-viscosity model is apparent in the underestimation of the standard deviation of the SFS stresses and the inability to predict transfers of kinetic energy from the subfilter scales to the resolved scales. Overall, the mixed model is found to have the best performance.