Second harmonic generation from plasmonic nanoantennas is investigated numerically using a surface integral formulation for the calculation of both the fundamental and the second harmonic electric field. The comparison between a realistic and an idealized gold nanoantenna shows that second harmonic generation is extremely sensitive to asymmetry in the nanostructure shape even in cases where the linear response is barely modified. Interestingly, minute geometry asymmetry and surface roughness are clearly revealed by far-field analysis, demonstrating that second harmonic generation is a promising tool for the sensitive optical characterization of plasmonic nanostructures. Furthermore, defects located where the linear field is strong (e.g., in the antenna gap) do not necessarily have the strongest impact on the second harmonic signal.