We show experimentally and by comparison with theory that during particle size-segregation in a sheared granular flow large particles rise slower in regions of many small particles and small particles sink faster in regions of many large particles. Binary mixtures with increasing amount of small particles take longer to fully segregate and vice versa. In addition, the saturation of the bottom of the flow with small particles is faster than saturation of large particles at the top of the flow. Our results, therefore, show that the segregation rates of the large and small particles have an asymmetric dependency on the local particle concentration. This has important repercussions for the modeling of size segregation which has up till now not taken into account this effect and considered symmetric dependency of segregation rates on concentration. This discovery draws parallels between the dynamics of size segregation and the processes of traffic flow, sedimentation and particle diffusion, which also exhibit asymmetric behavior.