Invasive lobular carcinoma (ILC) is the most frequent special histological subtype of breast cancer, typically characterized by loss of E‐cadherin. It has clinical features distinct from other estrogen receptor‐positive (ER+) breast cancers but the molecular mechanisms underlying its characteristic biology are poorly understood because we lack experimental models to study them. Here, we recapitulate the human disease, including its metastatic pattern, by grafting ILC‐derived breast cancer cell lines, SUM‐44 PE and MDA‐MB‐134‐VI cells, into the mouse milk ducts. Using patient‐derived intraductal xenografts from lobular and non‐lobular ER+ HER2− tumors to compare global gene expression, we identify extracellular matrix modulation as a lobular carcinoma cell‐intrinsic trait. Analysis of TCGA patient datasets shows matrisome signature is enriched in lobular carcinomas with overexpression of elastin, collagens, and the collagen modifying enzyme LOXL1. Treatment with the pan LOX inhibitor BAPN and silencing of LOXL1 expression decrease tumor growth, invasion, and metastasis by disrupting ECM structure resulting in decreased ER signaling. We conclude that LOXL1 inhibition is a promising therapeutic strategy for ILC.