This paper presents a wave-based numerical scheme based on a spectral element method, coupled with an implicit-explicit Runge-Kutta time stepping method, for simulating room acoustics in the time domain. The scheme has certain features which make it highly attractive for room acoustic simulations, namely (a) its low dispersion and dissipation properties due to a high-order spatio-temporal discretization; (b) a high degree of geometric flexibility, where adaptive, unstructured meshes with curvilinear mesh elements are supported; and (c) its suitability for parallel implementation on modern many-core computer hardware. A method for modelling locally reacting, frequency dependent impedance boundary conditions within the scheme is developed, in which the boundary impedance is mapped to a multipole rational function and formulated in differential form. Various numerical experiments are presented, which reveal the accuracy and cost-efficiency of the proposed numerical scheme.