This paper investigates the influence of the choice of the cost function in the optimal control formulation for an air-to-water heat pump system. The aim is to minimize, under given thermal comfort requirements, the electricity consumption which is calculated as the ratio between the thermal power and the coefficient of performance (COP) at which the heat is produced. Because the COP depends on the supply water temperature, which in turn depends on the thermal power, the resulting optimization problem is non-convex. This study compares the results obtained for the original, non-convex optimal control problem formulation to the results obtained for convex approximations typically found in the literature. The study reveals that these convex approximations yield almost identical results as the original, non-convex one, namely a smooth profile for the thermal power which is correlated to the ambient air temperature profile. This is an interesting result since the considered convex optimization problems can be solved faster to (global) optimality compared to the considered non-convex problems which in general require significantly more computational effort in order to reach global optimality.