Curvilinear structures are frequently observed in a variety of domains and are essential for comprehending neural circuits, detecting fractures in materials, and determining road and irrigation canal networks. It can be costly and time-consuming to manually extract these structures, so automated systems are required for faster and more accurate analysis. Recently deep learning-based approaches that rely solely on deep neural networks are being used for automatic delineation of such structures. However, preserving the topology of the curvilinear structures, which is essential for downstream applications, is a significant challenge. Furthermore, deep learning models require vast quantities of precisely annotated data, which is difficult to acquire, especially for 3D data. Commonly used pixel-wise loss functions cannot capture the topology of curvilinear structures and deep networks that are trained with such losses are prone to imprecisions in the annotations. In this thesis, we propose topology-aware loss functions to tackle these problems and improve the topology of the reconstructions.

We begin by introducing a connectivity-oriented loss function for extracting network-like structures from 2D images. We express the connectivity of curvilinear structures in terms of disconnections that they create between background regions of the image. Our loss function is designed to prevent such unwanted connections between background regions, and therefore close the gaps in predicted structures. Then, we focus on using Persistent Homology (PH) to improve the topological quality of the reconstructions. We propose a new filtration technique by fusing two existing approaches: filtration by thresholding and height function. With the proposed technique, we include location information of topological errors and increase the descriptive power of PH. In order to tackle imprecise annotations, we propose an active contour model based loss function. We treat annotations as contour models and allow them to deform themselves over correct centerlines during training while preserving the topology of the structure. Our final contribution is extending the aforementioned connectivity-oriented loss function to work with 3D data as well. We achieve this by computing the loss on 2D projections. With this method, we also reduce the annotation effort required to provide training data. We demonstrate, in experiments on 2D (satellite images of roads and irrigation canals) and 3D (Magnetic Resonance Angiography, Two-Photon Microscopy) datasets, that our loss functions significantly improve the topological quality of the reconstructions.