The JPEG committee (Joint Photographic Experts Group, formally known as ISO/IEC SC29 WG1) is currently in the process of standardizing JPEG XS, a new interoperable solution for low-latency, lightweight and visually lossless compression of image and video. This codec is intended to be used in applications where content would usually be transmitted or stored in uncompressed form such as in live production, display links, virtual and augmented reality, self driving vehicles or frame buffers. It achieves bandwidth and power reduction for transparent and low latency coding for compression ratios ranging from 2:1 to 6:1. The subjective assessment of the impact of visually lossless compression poses particular challenges. This paper describes the subjective quality evaluation conducted on the JPEG XS core coding system. In particular, it details the test procedures and compares the results obtained by the different evaluation laboratories involved in the standardization effort.