Phage display is a powerful tool for the selection of highly affine, short peptide ligands. While originally primarily used for the identification of ligands to proteins, the scope of this technique has significantly expanded over the past two decades. Phage display nowadays is also increasingly applied to identify ligands that selectively bind with high affinity to a broad range of other substrates including natural and biological polymers as well as a variety of low-molecular-weight organic molecules. Such peptides are of interest for various reasons. The ability to selectively and with high affinity bind to the substrate of interest allows the conjugation or immobilization of, e.g., nanoparticles or biomolecules, or generally, facilitates interactions at materials interfaces. On the other hand, presentation of peptide ligands that selectively bind to low-molecular-weight organic materials is of interest for the development of sensor surfaces. The aim of this article is to highlight the opportunities provided by phage display for the identification of peptide ligands that bind to synthetic or natural polymer substrates or to small organic molecules. The article will first provide an overview of the different peptide ligands that have been identified by phage display that bind to these "soft matter" targets. The second part of the article will discuss the different characterization techniques that allow the determination of the affinity of the identified ligands to the respective substrates.