The early visual system is organized retinotopically. However, motion perception occurs in non-retinotopic coordinates. Even though many perceptual studies revealed the central role of non-retinotopic processes, little is known about their neural correlates and mechanisms. Tadin and colleagues (2003) found that increasing the spatial size of a high-contrast drifting-Gabor deteriorates motion-direction discrimination whereas the opposite occurs with a low-contrast stimulus. This is proposed to reflect a perceptual correlate of an adaptive center-surround antagonism, whereby at low-contrast excitatory center dominates whereas at high-contrast suppressive-surround mechanisms become more effective. We tested the hypothesis that the non-retinotopic system also processes motion information by means of an adaptive center-surround mechanism. We used the Ternus-Pikler display, a paradigm that pits against each other retinotopic and non-retinotopic representations. The Ternus-Pikler display contained three Gabor-patches. Depending on ISI (133ms vs. 0ms), either group- or element-motion is perceived, i.e., either all Gabors moved back and forth in tandem or the utmost Gabors jumped alternating left-right. One of the Gabors in the display contained a fixed phase-shift that created the perception of coherent drift in either retinotopic or non-retinotopic coordinates. Observers were instructed to attend to one of the Gabors in the display and report its drift direction. We measured phase-shift thresholds for motion-direction discrimination while varying the size and contrast of the stimulus. Our results show a statistically significant interaction of size and contrast in both retinotopic and non-retinotopic tasks. We observed increases in thresholds as a function of size at high-contrast values and threshold decreases as a function of size at weak contrast values, thereby generalizing Tadin et al.’s results to non-retinotopic processing. Our results suggest that the non-retinotopic process may also be mediated by an adaptive center-surround mechanism where at low-contrast spatial summation prevails and then shifts to surround suppression as the input contrast increases.