At the conceptual stage, building performance simulation (BPS) based evaluations are being increasingly used for tasks such as ranking of competing massing design proposals. However, such conceptual stage evaluations suffer from information deficiency in building level design attributes. The resulting uncertainty in performance evaluations raises questions regarding their usefulness for decision-making. We used a risk-based decision evaluation metric called expected opportunity loss to assess the reliability of a BPS-based ranking of conceptual stage massing schemes. We found day-lighting assessments (spatial Daylight Autonomy) to be least reliable, with 22% chance of making an incorrect decision at the conceptual stage, followed by annual heating (15%) and cooling demand (8%). This work provides a structured framework for evaluating utility of conceptual stage BPS models and a purposeful basis for integration of BPS assessments in the design process, subject to level of design development.