Selective area epitaxy (SAE) provides the path for scalable fabrication of semiconductor nanostructures in a device-compatible configuration. In the current paradigm, SAE is understood as localized epitaxy, and is modelled by combining planar and self-assembled nanowire growth mechanisms. Here we use GaAs SAE as a model system to provide a different perspective. First, we provide evidence of the significant impact of the annealing stage in the calculation of the growth rates. Then, by elucidating the effect of geometrical constraints on the growth of the semiconductor crystal, we demonstrate the role of adatom desorption and resorption beyond the direct-impingement and diffusion-limited regime. Our theoretical model explains the effect of these constraints on the growth, and in particular why the SAE growth rate is highly sensitive to the pattern geometry. Finally, the disagreement of the model at the largest pitch points to non-negligible multiple adatom recycling between patterned features. Overall, our findings point out the importance of considering adatom diffusion, adsorption and desorption dynamics in designing the SAE pattern to create pre-determined nanoscale structures across a wafer. These results are fundamental for the SAE process to become viable in the semiconductor industry.