We introduce a new approach for finger-spelling recognition from video sequences, relying on the collaboration between the feature extraction and behavior inference processes. The inference process dynamically guides the segmentation- based feature extraction process towards the most likely location of the signer's hand (based on its attributes). Reciprocally, segmentation offers to the inference process hand object attributes extracted from each image, combining the received guidance and new image information. This collaboration is beneficial for both processes, yielding not only accurate segmentations of the spelling hand, but also a robust recognition scheme, which can cope with complex backgrounds, typical of real life situations.