The representation of video information in terms of its content is atthe foundation of many multimedia applications, such as broadcasting,content-based information retrieval, interactive video, remotesurveillance and entertainment. In particular, object-basedrepresentation consists in decomposing the video content into acollection of meaningful objects. This approach offers a broad rangeof capabilities in terms of access, manipulation and interaction withthe visual content. The basic difference when compared withpixel-based procedures is that instead of processing individualpixels, image objects are used in the representation. To exploit thebenefits of object-based representation, multimedia applications needautomatic techniques for extracting such objects from video data, aproblem that still remains largely unsolved. In this paper, we firstreview the extraction techniques that enable the separation offoreground objects from the background. Their field of applicabilityand their limitations are discussed. Next, automatic tools forevaluating their performances are introduced. The major applicationsthat benefit from an object-based approach are then analysed. Finally,we discuss some open research issues in object-based video.