Drones are nowadays considered as a valuable solution to monitor urban traffic. Object detectors face numerous challenges when dealing with high-resolution aerial images captured by drones, due to variations in altitude, viewing angle, and weather conditions. To address these challenges, we present an object detector called Butterfly detector that is tailored to detect objects in aerial images. It is an anchor-free method that leverages field-based representations. We introduce Butterfly fields, a type of composite field that describes the spatial information of output features as well as the scale of the detected objects. We employ a voting mechanism between related Butterfly vectors pointing to the object center. We highlight the benefits of our method for urban traffic monitoring by (i) evaluating the recall/precision rate of our detector on two publicly available drone datasets (UAVDT and VisDrone2019), and (ii) measuring the error rate for flow estimations on our newly released EPFL roundabout dataset. We outperform the performance of previous methods while remaining real-time.