After endocytic uptake, influenza viruses transit early endosomal compartments and eventually reach late endosomes. There, the viral glycoprotein hemagglutinin (HA) triggers fusion between endosomal and viral membrane, a critical step that leads to release of the viral segmented genome destined to reach the cell nucleus. Endosomal maturation is a complex process involving acidification of the endosomal lumen as well as endosome motility along microtubules. While the pH drop is clearly critical for the conformational change and membrane fusion activity of HA, the effect of intracellular transport dynamics on the progress of infection remains largely unclear. In this study, we developed a comprehensive mathematical model accounting for the first steps of influenza virus infection. We calibrated our model with experimental data and challenged its predictions using recombinant viruses with altered pH sensitivity of HA. We identified the time point of virus-endosome fusion and thereby the diffusion distance of the released viral genome to the nucleus as a critical bottleneck for efficient virus infection. Further, we concluded and supported experimentally that the viral RNA is subjected to cytosolic degradation strongly limiting the probability of a successful genome import into the nucleus.