Performing remote manipulation tasks by teleoperation with limited bandwidth, communication delays and environmental differences is a challenging problem. In this paper, we learn a task-parameterized generative model from the teleoperator demonstrations using a \textit{hidden semi-Markov model} that provides assistance in performing remote manipulation tasks. We present a probabilistic formulation to capture the intention of the teleoperator, and subsequently assist the teleoperator by time-independent shared control and/or time-dependent autonomous control formulations of the model. In the shared control mode, the model corrects the remote arm movement based on the current state of the teleoperator; whereas in the autonomous control mode, the model generates the movement of the remote arm for autonomous task execution. We show the formulation of the model with virtual fixtures and provide comparisons to benchmark our approach. Teleoperation experiments with the Baxter robot for reaching a movable target and opening a valve reveal that the proposed methodology improves the performance of the teleoperator and caters for environmental differences in performing remote manipulation tasks.