GPS devices allow recording the movement track of the moving object they are attached to. This data typically consists of a stream of spatio-temporal (x,y,t) points. For application purposes the stream is transformed into finite subsequences called {\em trajectories}. Existing knowledge extraction algorithms defined for trajectories mainly assume a specific context (e.g. vehicle movements) or analyze specific parts of a trajectory (e.g. stops), in association with data from chosen geographic sources (e.g. points-of-interest, road networks). We investigate a more comprehensive semantic annotation framework that allows enriching trajectories with any kind of semantic data provided by multiple 3rd party sources. This paper presents SeMiTri - the framework that enables annotating %all meaningful parts of trajectories for any kind of moving objects. Doing so, the application can benefit from a {\it \lq\lq semantic trajectory\rq\rq} representation of the physical movement. The framework and its algorithms have been designed to work on trajectories with varying data quality and different structures, with the objective of covering abstraction requirements of a wide range of applications. Performance of SeMiTri has been evaluated using many GPS datasets from multiple sources -- including both fast moving objects (e.g. cars, trucks) and people's trajectories (e.g. with smartphones). These two kinds of experiments are reported in this paper.