The motional sideband asymmetry of a mechanical oscillator interacting with a laser field can be observed when approaching the quantum ground state, where the zero-point energy of the mechanical oscillator becomes a sizable contribution to its motion. In the context of quantum optomechanics, it allows, in principle, calibration-free inference of the thermal equilibrium of a macroscopic mechanical resonator with its optical bath. At room temperature, this phenomenon has been observed in pioneering experiments using levitated nanoparticles. Measuring this effect with solid-state mechanical resonators has been compounded by thermal intermodulation noise, mirror frequency noise and low quantum cooperativity. Here, we sideband-cool a membrane-in-the-middle system close to the quantum ground state from room temperature and observe motional sideband asymmetry in a dual-homodyne measurement. Sideband thermometry yields a minimum phonon occupancy of n¯eff=9.5. Our work provides insights into nonlinear optomechanical dynamics at room temperature and facilitates accessible optomechanical quantum technologies without the need for complex feedback control and cryogenic cooling.