The properties of the interface between solid and melt are key to solidification and melting, as the interfacial free energy introduces a kinetic barrier to phase transitions. This makes solidification happen below the melting temperature, in out-of-equilibrium conditions at which the interfacial free energy is ill defined. Here we draw a connection between the atomistic description of a diffuse solid-liquid interface and its thermodynamic characterization. This framework resolves the ambiguities in defining the solid-liquid interfacial free energy above and below the melting temperature. In addition, we introduce a simulation protocol that allows solid-liquid interfaces to be reversibly created and destroyed at conditions relevant for experiments. We directly evaluate the value of the interfacial free energy away from the melting point for a simple but realistic atomic potential, and find a more complex temperature dependence than the constant positive slope that has been generally assumed based on phenomenological considerations and that has been used to interpret experiments. This methodology could be easily extended to the study of other phase transitions, from condensation to precipitation. Our analysis can help reconcile the textbook picture of classical nucleation theory with the growing body of atomistic studies and mesoscale models of solidification.