Machine learning has experienced significant growth in the smart building sector, whether for building modeling or energy management. Data-driven approaches leverage available measurements to bypass the slow and costly calibration of physics-based models, offering adaptability, low maintenance and greater flexibility. However, the quality of these models depends on historical data, which may be lacking for newly constructed buildings. This paper introduces a fully data-driven modular approach, from temperature modeling to heating control, that requires few data when transferred from a source to a target building. The controller consists of two modules: a deep reinforcement learning agent that manages the desired room temperature and an action-mapper specific to each room that adjusts heating controls. To adapt the controller to a new room, only the action-mapper is substituted. This approach requires just a few weeks of data and reuses an effective policy with minimal effort. The controller is trained using a neural network-based environment simulator, incorporating physical consistency to ensure accurate states and rewards. Simulations and real-world tests show the modular controller achieves 13 % average energy savings (up to 17 %) compared to traditional transfer learning methods, and 26 % (up to 32 %) compared to rule-based controllers, without compromising comfort.