Semantic segmentation is generally associated with second generation video coders, or object-based coders. Object-based coders encode different video objects separately in order to achieve lower bitrates and to enable object-based functionalities. In this paper, we present an encoding framework that uses semantic segmentation to improve the performance of first generation video coders, or frame-based coders. Semantic segmentation is exploited in a prefiltering step prior to encoding. This prefiltering step mimics the way humans treat visual information by separating relevant information from contextual information. Contextual information is then simplified, thus reducing the information to be coded. Experimental results on indoor as well as on outdoor test sequences when the semantics is defined by motion show that the proposed prefiltering improves the perceived quality with respect to traditional video coders.