Visual working memory (VWM) is prone to interference from stored items competing for its limited capacity. Distinctiveness or similarity of the items is acknowledged to affect this competition, such that poor item distinctiveness causes a failure to discriminate between items sharing common features. In three experiments, we studied how the distinctiveness of studied real-world objects (i.e., whether the objects belong to the same or different basic categories) affects the retrieval of objects themselves (simple recognition) and object-location conjunctions (information about which object was where in a display, cued recall). In Experiments 1 and 2, we found that distinctiveness did not affect memories for objects or for locations, but lowdistinctive objects were more frequently reported at “swapped” locations that originally contained other objects, showing objectlocation memory swaps. In Experiments 3 we found that observers swapped the location of a tested object with another object from the same category more frequently than with any of the objects from another category. This suggests that more similar studied objects cause more retrieval competition in object-location judgments than in simple recognition. Additionally, we discuss a possible role of categorical labeling of locations that can support object-location retrieval when the studied objects are highly distinct.