In light of the critical threat to coral reefs worldwide due to human activity, innovative monitoring strategies are demanded that are efficient, scalable, and low-cost. Existing methods are commonly very labour intensive, creating a bottleneck for large-scale applicability. We present a method for rapid 3D semantic mapping from video transects using low-cost underwater cameras, in which both the 3D reconstruction and the benthic classification are created by neural networks trained on field data. The 3D reconstruction neural network is trained in a self-supervised scheme from a large dataset of videos, learning to tackle the challenges that conventional computer vision methods face under water. To train the semantic segmentation neural network, we create a dataset of annotated video frames with over 80’000 polygons from 36 benthic classes, down to the resolution of prominent visually identifiable coral genera found in the shallow reefs of the Red Sea. This initiative was carried out in Djibouti, Sudan, Jordan, and Israel, with over 150 hours of collected video footage for training the neural network for 3D reconstruction. We then process over 30 transects using the deep-learning based mapping system, characterizing their benthic composition, and demonstrate the method's accuracy as well as the consistency over time (evaluated after re-visiting sites in Israel and Djibouti). This research pioneers the use of deep learning in 3D underwater mapping and semantic segmentation for rapid, practical, and implementable reef surveying from cheap underwater cameras, paving the way for affordable and widespread deployment of the method in monitoring.