Traffic forecasting has been a fundamental task in transportation research, with many methods and datasets mainly based on highway loop detector data. In recent years, drones have become a favorable choice for urban traffic monitoring, due to their flexibility, high data quality and larger spatial coverage. Although drones have promising possibilities of becoming an additional asset in traffic monitoring, the lack of public datasets and methods has made the joint use of drone and loop detector data fairly under-explored. Therefore, we create a novel multi-source dataset SimBarca from simulated vehicle trajectories for urban traffic forecasting, featuring speed measurements from both drones and loop detectors. Additionally, we provide a graph-based model HiMSNet to handle multiple input modalities and evaluate it along with other basic benchmark predictors. Our analysis shows that HiMSNet achieves good performance for regional speed prediction, and outperforms the baselines for road segment-level speed prediction. Experiments with various configurations on data modality, sensor coverage and noises have demonstrated the great potential of urban traffic forecasting with multi-source data.