After an earthquake, the structural safety of all buildings in the affected region needs to be assessed. This is currently done by a visual inspection of each building. This approach is very time-consuming and requires a significant amount of expertise from the inspectors. A lack of the latter can lead to inaccurate assessments, which are often overly conservative. This is of course, on the safe side but negatively affects the resilience of societies. In the future, the post-earthquake assessment process will be accelerated and objectified by replacing the visual inspection with an automated image-based evaluation of the buildings. One research line of the Earthquake Engineering and Structural Dynamics Laboratory is to develop such a process for stone masonry buildings, which are among the most vulnerable buildings during earthquake loading. The objective of this thesis is to contribute to this vision. Earthquake damage to stone masonry structures manifests itself mainly in cracks. Surface cracks have been shown to contain important information to build empirical and physics-based models to understand the activated mechanisms and estimate the level of damage to a structure. This Ph.D. thesis aims at developing methods for detecting, segmenting, quantifying, and interpreting surface cracks using optical measurement and machine learning techniques. The methods are applied to a set of six large-scale quasi-static cyclic shear-compression tests on plastered rubble stone masonry piers, which have been conducted as part of the thesis and provide a unique data set of high-resolution images and displacement-field measurements. The output is used (i) to investigate the influence of shear span ratio, axial load ratio, and the number of cycles on the crack width and (ii) to build empirical models for the stiffness loss, the residual force, and the drift capacity of rubble stone masonry piers as a function of crack features.