Perception of an element can be strongly modulated by surround. Likewise, neural firing of a neuron can strongly be influenced by stimuli outside the classical receptive field. Contextual modulation is often seen as a limitation of the visual system, where nearby flankers impede spatial resolution and thus deteriorate performance on target elements. Accordingly, contextual modulation is usually explained by models, in which only nearby elements influence each other. Here, I will show that perception of an element depends on all elements in the entire visual field. Contextual modulation is rather an interpretation than a limitation. Performance on a target depends on how it groups with all elements in the entire visual field. Performance is good when the target ungroups from the context and deteriorates when it groups with the context.