Most visual search models rely on a retinotopic saliency map. Here, we provide evidence that visual saliency is computed non-retinotopically. Recently, it was shown that attention can operate in a non-retinotopic reference frame by inserting a search display in a Ternus-Pikler display [Boi et al, 2009 Journal of Vision 9(13):5, 1–11]. From one frame to another, the display moved back and forth, producing an apparent motion percept. Here, we present an experiment in which the target (a vertically oriented pair of dots) and three distractors (horizontally oriented pairs of dots) were superimposed on non-informative shapes (small red diamonds and green disks). At each frame, the red diamonds turned to green disks and vice versa, while the dot pairs stayed the same. These changes, which occurred in non-retinotopic coordinates, captured attention. As a result, performance in the search task was deteriorated compared to a control condition in which the shapes did not change. Hence, non-retinotopic task-irrelevant stimulus saliency slowed down the search process, arguing for a non-retinotopic saliency map.