While understanding the shear strength of stone masonry structures is important for the design and the mainte- nance, we still lack computational tools for predicting the strength as a function of the stone layout. Here we implement an end-to-end image based kinematic analysis framework that converts the image of a stone layout of a wall into a 2D kinematic model. Machine learning and image processing techniques are applied to convert a wall image into a rigid block model, which is then used as the geometry input for an existing limit analysis approach using mathematical pro- gramming. This existing approach is extended such that also cohesion, limited tensile and compressive strength can be considered in the point-based formulation of interface failure. We apply the method to simulate the strength of stone masonry walls with mortar that are subjected to shear-compression loading and show that our method can demonstrate the influence of the stone masonry typology on the shear strength.