In this paper, we advance the state of the art in variational image segmentation through the fusion of bottom-up segmentation and top-down classification of object behavior over an image sequence. Such an approach is beneficial for both tasks and is carried out through a joint optimization, which enables the two tasks to cooperate, such that knowledge relevant to each can aid in the resolution of the other, thereby enhancing the final result. In particular, classification offers dynamic probabilistic priors to guide segmentation, while segmentation supplies its results to classification, ensuring that they are consistent with prior knowledge. The prior models are learned from training data and updated dynamically, based on segmentations of earlier images in the sequence. We demonstrate the potential of our approach in a hand gesture recognition application, where the combined use of segmentation and classification improves robustness in the presence of occlusion and background complexity.