In construction robotics, a conventional design-to-fabrication work-flow starts with designing a structure, followed by task and robotic motion planning, and ultimately, fabrication. However, this approach can prove unsuccessful, as we may only discover the infeasibility of a design at the final stages of the process. This can result in rework and a considerable waste of time and resources. To overcome this challenge, we propose a design method based on reinforcement learning (RL) where the agent makes decisions at every step of the sequential assembly of the structure while considering assembly’s stability. In this way, we take the construction constraints into consideration at the design stage. The research particularly focuses on the design of spanning structures that multiple robot arms can construct without the need for scaffolding. A series of experiments were conducted using both a centralized and a decentralized learning setup. Our results show that while the decentralized setup was successful in constructing smaller structures, only the centralized setup allowed active collaboration between robot arms, resulting in structures with larger spans. To validate our approach, we fabricated two of the designed structures with two collaborating robot arms, which confirmed the feasibility of these designs. In summary, the proposed method opens exciting possibilities for generating innovative designs that push the boundaries of architectural creativity while simultaneously fulfilling fabrication-related constraints.