A novel analog front-end architecture based on complex sub-Nyquist sampling for the intermediate frequency (IF) stage of a mmWave receiver is proposed. With this front-end, the use of a wideband hybrid coupler and two half-rate analog-to-digital converters (ADCs) allow for a flexible placement of the IF. It is shown that digital compensation of the impairments introduced by the non-ideal 90 degree hybrid coupler is required to use high modulation orders. Further a digital signal processing (DSP) architecture is presented which performs equalization of a fractionally spaced sub-sampled IF signal in frequency domain (FD) and integrates the compensation of the impairments with low overhead. Based on this DSP architecture a working 60GHz single-carrier link is demonstrated. Measurement results show the feasibility of 256QAM modulated transmission with a bandwidth of up to 1.8 GHz and a resulting raw data rate of 12.8 Gb/s using our frontend architecture with the digital FD compensation.