Organoids are a new class of biological model systems that have garnered significant interest in the life sciences. When provided with the proper 3D matrix and biochemical factors, stem cells can self-organize and form tissue-specific organoids. Thus far, there has been a substantial effort to identify soluble niche components essential for organoid culture; however, the role of the solid extracellular matrix (ECM) as an essential element of the niche is still largely lacking. In this review, we discuss the importance of the ECM in intestinal, hepatic, and pancreatic organoid culture and how biomaterial-based approaches can be used to probe different ECM properties required for more physiologically and translationally relevant organoid models.