The potential of graphene for photonic applications was evidenced by recent demonstrations of modulators, polarization rotators and isolators. These promising yet preliminary results raise crucial questions: what is the optimal performance achievable by more complex designs and how can this optimum be achieved in practice? We answer by first demonstrating that the relevant figures of merit for the devices above are subject to absolute theoretical upper bounds. Strikingly, these limits are related only to the conductivity tensor of graphene; thus, we can provide essential roadmap information such as the best possible device performance versus wavelength and graphene quality. Second, based on the theory developed, physical insight and detailed simulations, we demonstrate how structures closely approaching these fundamental limits can be designed, demonstrating the possibility of significant improvement. These results are believed to be of paramount importance for the design of modulators, rotators and isolators using graphene or other two-dimensional materials.