Cultured human epidermal keratinocyte stem cells (holoclones) are crucial for regenerative medicine for burns and genetic disorders. In serial culture, holoclones progressively lose their proliferative capacity to become transient amplifying cells with limited growth (paraclones), a phenomenon termed clonal conversion. Although it negatively impacts the culture lifespan and the success of cell transplantation, little is known on the molecular mechanism underlying clonal conversion. Here, we show that holoclones and paraclones differ in their actin filament organization, with actin bundles distributed radially in holoclones and circumferentially in paraclones. Moreover, actin organization sets the stage for a differing response to epidermal growth factor (EGF), since EGF signalling induces a rapid expansion of colony size in holoclones and a significant reduction in paraclones. Furthermore, inhibition of PI3K or Rac1 in holoclones results in the reorganization of actin filaments in a pattern that is similar to that of paraclones. Importantly, continuous Rac1 inhibition in holoclones results in clonal conversion and reduction of growth potential. Together, our data connect loss of stem cells to EGF-induced colony dynamics governed by Rac1.