Coral reefs, which host more than a third of the ocean’s biodiversity on less than 0.1% of its surface, are existentially threatened by climate change and other human activities. This necessitates methods for evaluating the state of coral reefs that are efficient, scalable, and low-cost. Current digital reef monitoring tools typically rely on conventional Structure-from-Motion photogrammetry, which can limit the scalability, and current datasets for training semantic segmentation systems are either sparsely labeled, domain-specific, or very small. We describe the first deep-learning-based 3D semantic mapping approach, which enables rapid mapping of coral reef transects by leveraging the synergy between self-supervised deep learning SLAM systems and neural network-based semantic segmentation, even when using low-cost underwater cameras. The 3D mapping component learns to tackle the challenging lighting effects of underwater environments from a large dataset of reef videos. The transnational data-collection initiative was carried out in Djibouti, Sudan, Jordan, and Israel, with over 150 hours of collected video footage for training the neural network for 3D reconstruction. The semantic segmentation component is a neural network trained on a dataset of video frames with over 80’000 annotated polygons from 36 benthic classes, down to the resolution of prominent visually identifiable genera found in the shallow reefs of the Red Sea. This research paves the way for affordable and widespread deployment of the method in analysis of video transects in conservation and ecology, highlighting a promising intersection with machine learning for tangible impact in understanding these oceanic ecosystems.