The response of simple plasmonic nanorods to polarized illumination is studied in detail. Depending on the orientation of that polarization with respect to the symmetry axes of the nanostructure, a chiral response can occur, which can be analyzed through a second polarizer, in order to control the spectral response of the system. Specifically, for the Ag nanorods fabricated here, a broad variety of colors can be produced that cover half of the chromaticity diagram. Depending on the illumination and detection polarizations, these colors range from white to vivid colors or even black, in spite of the fact that the material at hand does not absorb much light. By exploiting two additional degrees of freedom, namely the nanorod length and its orientation within the unit cell, it is possible to produce a very rich palette of optical effects that are controlled by the polarization of light. Their utilization to reproduce artworks is demonstrated, together with their operation as encrypting system, where the polarizations are used as keys and the message is encrypted in a quaternary color subset.