Although graphene has met many of its initially predicted optoelectronic, thermal, and mechanical properties, photodetectors with large spectral bandwidths and extremely high frequency responses remain outstanding. In this work, we demonstrate a >500 gigahertz, flat-frequency response, graphene-based photodetector that operates under ambient conditions across a 200-nanometer-wide spectral band with center wavelengths adaptable from <1400 to >4200 nanometers. Our detector combines graphene with metamaterial perfect absorbers with direct illumination from a single-mode fiber, which breaks with the conventional miniaturization of photodetectors on an integrated photonic platform. This design allows for much higher optical powers while still allowing record-high bandwidths and data rates. Our results demonstrate that graphene photodetectors can outperform conventional technologies in terms of speed, bandwidth, and operation across a large spectral range.