Augmented reality in additive fabrication is predominantly applied to the manufacturing of structures from regularly shaped materials. Our work however embraces natural heterogeneity, by focusing on the assembly of irregularly shaped elements such as mineral scraps. We introduce a computer-aided framework with a 3D stacking engine and an augmented reality interface capable of guiding users through the balanced positioning of discrete and highly nonuniform objects according to an on-the-fly computed model and without prior trimming of the building unit. The pipeline is tested by the construction of two dry-stone structures (i.e., lacking any adhesives), and its accuracy is validated with a comparative study between the point cloud of the as-built digitized artifacts and the generated execution model, which shows an average 2.9 ± 1.8 cm error between the stones of the two models. We finally show how the proposed framework could be improved both in terms of software and hardware. In the interests of reproducibility, all the methods are shared as open source with the community.