We present an eye-tracking study centered on the shift of focus from digital to physical layer in case of a handheld magic-lens system. In particular, we investigate which variables might affect the occurrence of such events. The experiment involved 35 participants who used a tablet AR application to explore structural behaviour through problem solving exercises. The tablet augmented the physical model of a structure with the information about the applied loads and the internal forces. Participants were asked to identify the nature of some forces by visual inspection. During the task they could freely navigate around the structure and touch it. Our results suggest that the amount of shifts to real-world objects is related to the user's spatial orientation rather than on the user's spatial skills or the digital augmentation flaws. Specifically, the probability of looking at the physical layer increases when rapid movements are performed. The task difficulty and the perceived controllability, although not influencing the number of shifts, affected the mental effort during such moments.