After more than a century of research, the mechanisms underlying visual masking are still hotly debated. One key characteristic of masking is that variations in the stimulus onset asynchrony (SOA) between the target and the mask can lead to either monotonic reductions in the effect of the mask on the target (A-type masking) or an increase in masking for intermediate SOAs and then a decrease in masking for longer SOAs (B-type masking). Past experimental and theoretical work suggested that the type of the masking function depends on the strength of the mask relative to the target. Usually, mask strength is related to energy (stimulus intensity x duration). Here, we show that the overall spatial layout of the mask is a much stronger factor than classical energy to explain the type of masking function.