Plastic litter is a major environmental hazard that endangers human, animal, and plant health on the planet. A substantial portion of plastic pollutants is washed from rivers and beaches into the oceans and aggregates at the surface as marine debris before decomposing into microplastics and being digested by animals or sedimented on the sea floor. The marine debris is inherently challenging to annotate manually on satellite images, as the boundaries of floating objects are not sharp, and a specific mixture of water is always present at the pixel level. Hence, all available annotated marine debris datasets suffer from annotation errors. In this work, we present a label refinement algorithm for marine debris detection that improves upon rough hand annotations and considers the spectral characteristics of marine debris. We show quantitatively that a deep learning model trained with improved annotations achieves a higher classification accuracy on confirmed marine debris on two out of three datasets of confirmed plastic marine debris in Africa (in Ghana and South Africa). Thanks to the refinement module, we improve results for an environmentally important application that would benefit from further research attention to mitigate important associated challenges like label noise, domain shifts, and severe class imbalance.