This paper presents a new workflow for the generation of color and luminance calibrated high dynamic range (HDR) images. By investigating the relationship between some of the key generation steps along the path from light entering the camera to a pixel in the final image, the important sources of uncertainty are identified and mitigated as much as possible. This paper shows how adjustments to the HDR pipeline and additional calibration steps will lead to more consistent results that are robust across a wide range of broad-spectrum lighting conditions. These adjustments include: correctly setting the saturation point according to the raw sensor values, using a pre-calibrated shutter speed correction, merging to account for sensor-blooming and photon noise, demosaicing after merging to preserve details and avoid artifacts, and applying a calibrated color transform. Then, to cope with lens flare, one of the major sources of inaccuracies in capturing HDR images of scenes where the direct sun is visible, a novel process that employs a camera mounted shadowband is introduced. To demonstrate the practical application of this workflow, a pilot data set of hemispheric facade environment maps suitable for use in daylight simulation is presented, involving sky and sun illumination with highly detailed scenes of varied view content. This pilot data set of HDR environment maps is validated against chroma/illuminance and pyranometer measurements achieving a mean absolute percentage error of 6.2% from reference illuminance measurements with an average CIELUV chromaticity difference less than one-fifth of a just noticeable difference.