In realistic urban environments, where buildings experience variable heights, the tallest structures have a disproportional impact on drag, mean flow and turbulence. Although wind-engineering studies document well the effects of individual high-rise buildings on the immediate surrounding, the impact of varying building heights on the larger horizontally averaged flow has not been quantified systematically for realistic urban configurations. We use Large Eddy Simulation (LES) as a mean to study the fully developed turbulent flow over and within a 512 x 512m subset of truthful urban geometry in the city of Basel, Switzerland. A periodic LES domain is centered on the location of a tower, where measurements of turbulence were performed in 2001/02, which allows a direct validation of the LES at a specific location in the domain. The Lagrangian scale-dependent LES model is adopted to parametrize the subgrid stresses in the bulk of the flow and buildings are taken into account adopting a discrete-forcing-approach immersed boundary method (IBM), with the geometry taken from a highly accurate digital building model. A series of high-resolution LES runs are performed for various directions of the approaching flow, and with all buildings included and then buildings above a certain height threshold progressively removed, to isolate the impact of tall structures. Results show how the presence of isolated tall buildings strongly modifies the roughness properties of the entire urban roughness sublayer, causing an increase in resolved pressure forces, which contributes to several percentages of the overall surface induced drag. In presence of tall buildings the local structure of the roughness sublayer is partitioned in two regimes: fine scale wake turbulence and elongated, high speed streak-like motions, locked between the position of isolated structures, with their axis aligned in the streamwise direction. For arrays with variable building heights statistics significantly differ from standard surface layers’ ones and are strongly inhomogeneous in space. Dispersive momentum fluxes are shown to significantly increase with the presence of isolated tall buildings, mainly due to the spatial heterogeneity of the flow in the spanwise direction.