We show that maximally helical hypermagnetic fields produced during pseudoscalar inflation can generate the observed baryon asymmetry of the Universe via the B + L anomaly in the Standard Model. We find that most of the parameter space of pseudoscalar inflation that explains the cosmological data leads to baryon overproduction; hence, the models of natural inflation are severely constrained. We also point out a connection between the baryon number and topology of the relic magnetic fields. Both the magnitude and sign of magnetic helicity can be detected in future diffuse gamma ray data. This could hint to a link between inflation and the baryon asymmetry of the Universe.