Light adaptation is crucial for coping with the varying levels of ambient light. Using high-density electroencephalography (EEG), we investigated how adaptation to light of different colors affects brain responsiveness. In a within-subject design, sixteen young participants were adapted first to dim white light and then to blue, green, red, or white bright light (one color per session in a randomized order). Immediately after both dim and bright light adaptation, we presented brief light pulses and recorded event-related potentials (ERPs). We analyzed ERP response strengths and brain topographies and determined the underlying sources using electrical source imaging. Between 150 and 261 ms after stimulus onset, the global field power (GFP) was higher after dim than bright light adaptation. This effect was most pronounced with red light and localized in the frontal lobe, the fusiform gyrus, the occipital lobe and the cerebellum. After bright light adaptation, within the first 100 ms after light onset, stronger responses were found than after dim light adaptation for all colors except for red light. Differences between conditions were localized in the frontal lobe, the cingulate gyrus, and the cerebellum. These results indicate that very short-term EEG brain responses are influenced by prior light adaptation and the spectral quality of the light stimulus. We show that the early EEG responses are differently affected by adaptation to different colors of light which may contribute to known differences in performance and reaction times in cognitive tests.