Transportation systems often rely on understanding the flow of vehicles or pedestrian. Fromtraffic monitoring at the city scale, to commuters in train terminals, recent progress in sensingtechnology make it possible to use cameras to better understand the demand,i.e., better trackmoving agents (e.g., vehicles and pedestrians). Whether the cameras are mounted on drones,vehicles, or fixed in the built environments, they inevitably remain scatter. We need to developthe technology to re-identify the same agents across images captured from non-overlappingfield-of-views, referred to as the visual re-identification task. State-of-the-art methods learn aneural network based representation trained with the cross-entropy loss function. We arguethat such loss function is not suited for the visual re-identification task hence propose to modelconfidence in the representation learning framework. We show the impact of our confidence-based learning framework with three methods: label smoothing, confidence penalty, and deepvariational information bottleneck. They all show a boost in performance validating our claim.Our contribution is generic to any agent of interest,i.e., vehicles or pedestrians, and outperformhighly specialized state-of-the-art methods across 6 datasets. The source code and models areshared towards an open science mission.