Cell fate progression of pluripotent progenitors is strictly regulated, resulting in high human cell diversity. Epigenetic modifications also orchestrate cell fate restriction. Unveiling the epigenetic mechanisms underlying human cell diversity has been difficult. In this study, we use human brain and retina organoid models and present single-cell profiling of H3K27ac, H3K27me3 and H3K4me3 histone modifications from progenitor to differentiated neural fates to reconstruct the epigenomic trajectories regulating cell identity acquisition. We capture transitions from pluripotency through neuroepithelium to retinal and brain region and cell type specification. Switching of repressive and activating epigenetic modifications can precede and predict cell fate decisions at each stage, providing a temporal census of gene regulatory elements and transcription factors. Removing H3K27me3 at the neuroectoderm stage disrupts fate restriction, resulting in aberrant cell identity acquisition. Our single-cell epigenome-wide map of human neural organoid development serves as a blueprint to explore human cell fate determination.|The mechanisms underlying human cell diversity are unclear. Here the authors provide a single-cell epigenome map of human neural organoid development and dissect how epigenetic changes control cell fate specification from pluripotency to distinct cerebral and retina neural types.