In this paper a procedure for subjective evaluation of the new JPEG XR codec for compression of still pictures is described in details. The new algorithm has been compared to the existing JPEG and JPEG 2000 standards when considering compression of high resolution 24 bpp pictures, by mean of a campaign of subjective quality assessment tests which followed the guidelines defined by the AIC JPEG ah-hoc group. Sixteen subjects took part in experiments at EPFL and each subject participated in four test sessions, scoring a total of 208 test stimuli. A detailed procedure for statistical analysis of subjective data is also proposed and performed. The obtained results show high consistency and allow an accurate comparison of codec performance.