The purpose of this paper is to identify the problems to solve in order to simulate real-time crowds in a Virtual Environment. We try to classify these problems and study how they have been addressed until now by the research community and our Lab in particular. We then discuss for each problem what are the,future challenges and how to address them.