Traditionally, time difference of arrival (TDOA) based acoustic source tracking consists of two stages, more precisely, estimation of TDOAs followed by a tracking algorithm. In general, these two stages are performed separately and presume that (1) TDOAs can be estimated reliably; and (2) the errors in detection behave in a well-defined fashion. The presence of noise and reverberation, however, leads to multimodal TDOA distributions and causes larger errors in the estimates, which ultimately lowers the tracking performance. To counteract this effect, we propose an approach that enhances TDOA estimation by (1) accounting for the multimodal aspect through a Gaussian mixture model and (2) integrating knowledge that has been obtained in the tracking stage. In doing so, this approach tightly couples the two stages. Experimental results on the AV16.3 corpus show that the proposed approach significantly improves the tracking performance compared to various other tracking algorithms.