This paper gives an overview of a vision-based wearable computer system ’SNAP&TELL’, which performs real-time gesture tracking for recognizing objects in the scene including outdoor landmarks. Our system uses a single camera to capture images which are processed using several algorithms to perform segmentation based on color, fingertip shape analysis, robust tracking, and invariant object recognition, in order to quickly identify the objects encircled by a user’s pointing gesture. In turn, the system returns information concerning the object such as its classification, historical facts,etc.This system provides enabling technology for the design of intelligent assistants to support ”Web-On-The-World” applications, with potential uses such as travel assistance, business advertisement, the design of smart living and working spaces, and pervasive wireless services and internet vehicles