We address the problem of modeling complex target behavior using a stochastic model that integrates object dynamics, statistics gathered from the environment and semantic knowledge about the scene. The method exploits prior knowledge to build point-wise polar histograms that provide the ability to forecast target motion to the most likely paths. Physical constraints are included in the model through a ray-launching procedure, while semantic scene segmentation is used to provide a coarser representation of the most likely crossable areas. The model is enhanced with statistics extracted from previously observed trajectories and with nearly-constant velocity dynamics. Information regarding the target's destination may also be included steering the prediction to a predetermined area. Our experimental results, validated in comparison to actual targets' trajectories, demonstrate that our approach can be effective in forecasting objects' behavior in structured scenes.