This study investigates patient's perspective on the use of AI in healthcare and the role of Explainable AI in this context. Through a co-creative workshop with six participants from diverse disciplines, we investigated the impact of transparency on trust. The findings highlight parallels between AI and doctors as “black boxes,” the complexity of informed consent and the importance of emotional safety. This work serves as a starting point for ongoing research that engages diverse stakeholder groups, to ensure the development of usercentered XAI solutions that can be effectively implemented in clinical practice.