Performance on a target can be modified by contextual elements. For example, when a Vernier is flanked by two lines, offset discrimination deteriorates compared to an unflanked presentation. This contextual modulation is usually explained by local spatial interactions between the Vernier and the flanks. We have shown that this explanation is inadequate. Instead, we propose that flanks interfere most strongly when they are grouped with the Vernier. In a number of experiments, we have shown that interference between flanks and Vernier can be reduced when the flanks form an independent group and the Vernier stands out. Our experiments show that this ungrouping can be established by differences in features like contrast polarity, colour, and stereoscopic depth, as well as flank grouping by figural properties. For example, Vernier discrimination thresholds are significantly lower when the Vernier is flanked by two cuboids compared to being flanked by two lines which are contained in the cuboids. Our results show that global Gestalt aspects are crucial in contextual modulation