Physical impairments like IQ imbalance and phase noise can cause significant degradation in the performance of wireless communication systems. In this paper, the joint effects of IQ imbalance and phase noise on OFDM systems are analyzed, and a compensation scheme is proposed to improve the system performance in the presence of IQ imbalance and phase noise. The scheme consists of a joint estimation of channel and impairment parameters and a joint data symbol estimation algorithm. It is shown both by theory and computer simulations that the proposed scheme can effectively improve the signal-to-noise ratio at the receiver. As a result, the sensitivity of OFDM receivers to the physical impairments can be significantly lowered, simplifying the RF and analog circuitry design in terms of implementation cost, power consumption, and silicon fabrication yield.