In this article, we present a multi-level time scales framework for the analysis of human-robot interaction (HRI). Such a framework allows HRI scientists to model the inter-relation between measures and factors of an experiment. Our final goal with the introduction of this framework is to unify scientific practice in the HRI community for better reproducibility. Our new approach transposes Newell’s framework of human actions to model human-robot interaction. Measures from the interaction are sorted into categories (time scales) corresponding to the temporal constraints proposed by Newell. According to this sorting, a bottom-up or top-down analysis can then be performed to correlate variables which allows a better understanding and explanation of the interaction. The utilization of our method within two experimental use cases is then presented. The first one, a child-robot interaction, involves two robots and one child playing a memory game. The second is based on an analysis of the PInSoRo dataset, involving 30 child-robot pairs in a freeplay interaction. Finally, we introduce clear guidelines to re-use the framework.