Detecting people from 2D images and analyzing their motion in 3D have been long standing computer vision problems central to numerous applications such as autonomous driving and athletic training. Recently, with the availability of large amounts of training data and the advent of deep learning, the performance in human segmentation, 3D human pose prediction has improved significantly. However, these problems remain challenging due to several factors. In this thesis, we decompose the human motion analysis into three sub-tasks; 2D human segmentation, 3D human body pose estimation and 3D human motion forecasting. Our goal is to alleviate the challenges in these problems using various encoder-decoder models.