Deep convolutional neural networks have shown remarkable results on face recognition (FR). Despite their significant progress, the performance of current face recognition techniques is often assessed in benchmarks under not always realistic conditions. The impact of outdoor environment, post-processing operations, and unexpected human behaviors are not sufficiently studied. This paper proposes a universal methodology that systematically measures the impact of various types of influencing factors on the performance of FR methods. Based on extensive experiments and analysis, the key influencing factors are identified, highlighting the need for suitable precautions on modern FR systems. The robustness of the state-of-the-art deep face recognition techniques is further benchmarked with our assessment framework. The best-performing CNN architecture and discriminative loss function are identified, in order to better guide the deployment of an FR system in real world.