The virtual orchestra is a multimodal system architecture based on complex 3D sound environment processing and a compact user interface. The system architecture supports high-performance sound content for VR (virtual reality) applications, and the user controls the performance with a magnetically tracked handheld device. The objective of this work is to give music amateurs the opportunity to conduct a group of musicians and produce a new kind of multimedia experience. To explore this idea, the authors developed a semi-immersive virtual reality system, based on a large projection screen, designed to create an entertaining multimedia experience for nonexpert users. They decided to implement a music performance simulator, a VR system where several virtual characters play different musical instruments and adapt their performance according to the user's instructions