Sequence modeling for signs and gestures is an open research problem. In thatdirection, there is a sustained effort towards modeling signs and gestures as a se-quence of subunits. In this paper, we develop a novel approach to infer movementsubunits in a data-driven manner to model signs and gestures in the frameworkof hidden Markov models (HMM) given the skeleton information. This approachinvolves: (a) representation of position and movement information with measure-ment of hand positions relative to body parts (head, shoulders, hips); (b) modelingthese features to infer a sign-specific left-to-right HMM; and (c) clustering theHMM states to infer states or subunits that are shared across signs and updat-ing the HMM topology of signs. We investigate the application of the proposedapproach on sign and gesture recognition tasks, specifically on Turkish signs Hos-piSign database and Italian gestures Chalearn 2014 task. On both databases, ourstudies show that, while yielding competitive systems, the proposed approach leadsto a shared movement subunit representation that maintains discrimination acrosssigns and gestures.