Epilepsy is a chronic neurological disorder that affects a significant portion of the human population and imposes serious risks in the daily life. Despite advances in machine learning and IoT, small, non-stigmatizing wearable devices for continuous monitoring and detection in outpatient environments are not yet widely available. Part of the reason is the complexity of epilepsy itself, including highly imbalanced data, multimodal nature, and very subject-specific signatures. However, another problem is the heterogeneity of methodological approaches in research, leading to slower progress, difficulty in comparing results, and low reproducibility. Therefore, this article identifies a wide range of methodological decisions that must be made and reported when training and evaluating the performance of epilepsy detection systems. We characterize the influence of individual choices using a typical ensemble random-forest model and the publicly available CHB-MIT database, providing a broader picture of each decision and giving good-practice recommendations, based on our experience, where possible.