In the field of UAV object tracking, correlation filter based approaches have received lots of attention due to their computational efficiency. The methods learn filters by the ridge regression and generate response maps to distinguish the specified target from the background. An ideal filter can predict the object’s position in a new frame, and in turn, can backtrack the object in the past frames. However, the neglect of tracking reversibility in most methods limits the potential of using inter-frame information to improve performance. In this work, a novel bidirectional incongruity-aware correlation filter is presented based on the nature of tracking reversibility. The proposed method incorporates the response-based bidirectional incongruity, which represents the gap between the filters’ discriminative difference in the forward and backward tracking perspective caused by object appearance changes. It enables the filter not only to inherit the discriminability from previous filters but also to enhance the generalization capability to unpredictable appearance variations in upcoming frames. Moreover, a temporary block-based strategy is introduced to empower the filter accommodate more drastic object appearance changes and make more effective use of inter-frame information. Comprehensive experiments are conducted on three challenging UAV tracking benchmarks, including UAV123@10fps, DTB70, and UAVDT. Experimental results indicate that the proposed method has superior performance compared with the other 34 state-of-the-art trackers. Our approach permits real-time performance at ~46.8 FPS on a single CPU and is suitable for UAV online tracking applications.