Machine learning and deep learning in particular have made a huge impact in many fields of science and engineering. In the last decade, advanced deep learning methods have been developed and applied to remote sensing and geoscientific data problems extensively. Applications on classification and parameter retrieval are making a difference: methods are very accurate, can handle large amounts of data, and can deal with spatial and temporal data structures efficiently. Nevertheless, several important challenges need still to be addressed. First, current standard deep architectures cannot deal with long-range dependencies so distant driving processes (in space or time) are not captured, and they cannot cope with non-Euclidean spaces efficiently. Second, as other data-driven techniques, deep learning models do not necessarily respect physical or causal relations. Finally, deep learning models are still obscure and resistant to interpretability. Advances are needed to cope with arbitrary signal structures and data relations, physical plausibility and interpretability. This paper discusses about ways forward to develop new DL methods for the Earth sciences in all three directions.