The successful development of an intelligent text mining application requires the collaboration of two main stakeholders: subject matter experts and text miners. In this paper, we describe a new methodology, agile text mining to improve that collaboration. Agile text mining is characterized by short development cycles, frequent tasks redefinition and continuous performance monitoring through integration tests. We introduce Sherlok, a system supporting the development of agile text mining applications and present an application to extract mention of neurons from a very large corpus of scientific articles. The resulting code and models are publicly available.