Space heating controls in offices usually follow static schedules detached from actual occupancy, which results in energy waste by unnecessarily heating vacant offices. The uniqueness of stochastic occupancy profile and thermal response time of each office are two main challenges in hard-programming a transferrable control logic that can adapt space heating schedule to the occupancy profile. This study proposes a Reinforcement Learning-based control framework (called DeepValve) that learns by itself how to adapt the space heating schedule to the occupancy profile in each office to save energy while maintaining comfort. All the aspects of the proposed framework (design, training, hardware setup, etc.) are centered on ensuring that it can be implemented on many offices in practice. The methodology includes three main steps: training on a wide variety of simulated offices with real-world occupancy data, month-long tests on three simulated offices, and day-long experimental tests in an environmental chamber. Results indicate that the agent can quickly adapt to new offices and save energy (40% reduction in total temperature increment) while maintaining occupant comfort. The results highlight the importance of occupant-centric control in offices.