Observing touch being applied to another human's face enhances tactile perception for touch being applied to one's own face. This effect, termed the Visual Remapping of Touch (VRT), is maximal the greater the physical or conceptual similarity between observer and observed. An interesting possibility, however, is that even though the basic nature of the VRT is multisensory, a high cognitive level affinity from the observer toward the observed could modulate the VRT even in the face of decreased physical similarity. In the present study we manipulate the level of attractiveness of the avatars that participants observed being touched. By doing so, we either increased (attractive) or decreased (unattractive) the interpersonal judgment value toward the avatar, while always decreasing the physical semblance between the avatar shown and the original image. Results revealed that both for an avatar depicting oneself or a stranger, the VRT is present when touch is applied to an attractive, but not to an unattractive avatar. These findings suggest that basic multisensory effects, such as visuo-tactile interaction, are modulated by higher-level cognitive representations of the self and of others.