A system for automated annotation of images and videos points a mobile device towards an object of interest, such as a building or landscape scenery, for the device to display an image of the scene with an annotation for the object. An annotation can include names, historical information, and links to databases of images, videos, and audio files. Different techniques can be used for determining positional placement of annotations, and, by using multiple techniques, positioning can be made more precise and reliable. The level of detail of annotation information can be adjusted according to the precision of the techniques used. Required computations can be distributed in an annotation system including mobile devices, servers and an inter-connecting network, allowing for tailoring of annotated images to mobile devices of different levels of complexity. A trade-off can be taken into account between precision of annotation and communication cost, delay and/or power consumption. An annotation database can be updated in a self-organizing way. Public information as available on the web can be converted to annotation data.