3D image and video have become popular since they can enhance the Quality of Experience (QoE) by adding the depth dimension to the traditional 2D media. In order to design and optimize the human-centric 3D multimedia processing techniques, it is important to understand the QoE perceived by users for given contents, as well as ways 2D and 3D media affect them. The goal of this paper is twofold. First, to investigate perceived QoE of 2D and 3D multimedia contents based on subjective ratings, and second, to investigate the way in which 2D and 3D multimedia technologies affect physiological signals. In particular, we construct subject- independent classification systems to predict 2D versus 3D multimedia experiences based on electroencephalography (EEG) and peripheral physiological signals including heart rate and respiration, from which the potential of EEG for this purpose is demonstrated.