Currently, post-earthquake assessment is performed visually, which implies that inspectors visit the damaged buildings and, based on their expertise, assign a damage grade. Yet, visual inspection is subjective, time-consuming and difficult to document. In this work, we present a novel and end-to-end framework to automatically generate 3D geometric digital twins of masonry buildings containing damage information and its characterization allowing inspectors to make on-site decisions about the assets’ safety and the required interventions. Our developed pipeline not only significantly reduces inspection time, but it also reduces the subjectivity of decision making because of human judgment. We plan to implement this framework as real-time system to be used in a variety of portable devices to be applied in real-world scenarios.