The polarization sensitivity of optical resonant dipole antennas is investigated numerically using the Green's tensor technique. The electric field-intensity in the feed-gap of the antenna is calculated as function of the polarization of the incident field. A simple analytical model is proposed that matches the numerical data very well. While a very strong polarization sensitivity can be achieved for specific wavelengths, our results also indicate that there are situations where the antenna is not sensitive at all to the polarization. The role played by different plasmon resonances in the system is illustrated. [DOI: 10.2971/jeos.2008.08018]