Aerial robot swarms can have a large socio-economic impact. They can perform time-critical missions faster than a single robot and access dangerous environments without compromising human safety. However, swarm deployment is often limited to free environments where no obstacle interferes with the robots' flight. Their trajectories are computed before the flight in many applications, and the robots are perceptually blind to one another. In these cases, all decisions are taken by a central computer that is informed about all robots' states, thus increasing their chance of failure in case of signal interruptions, limiting their robustness to failure, and preventing their scalability in size. Drones should have the autonomy to make their own decisions based on local information, and they should integrate a safe obstacle avoidance behavior to offer more versatility for their application in real-world environments. We propose a predictive approach to swarm control inspired by flocking birds to address these limitations. In particular, we develop methods that enable drones to coordinate their motion by predicting their future trajectory and coordinating with their neighbors based on local information. We first propose a centralized predictive algorithm that computes the trajectories of the drones in real-time and improves the synchronization and order of the swarm flight compared to current state-of-the-art algorithms. We show in extensive simulations that our algorithm is robust to different obstacle densities, swarm speeds, and inter-agent distances. Then, we formulate a distributed predictive algorithm with the same qualitative advantages as its centralized counterpart but scalable in the swarm size. We also show that our approach is tolerant to a wide range of sensor noise. Finally, we extend and analyze the usage of this algorithm also for pure sensor-based swarms that cannot use explicit communication. We validate the algorithms in extensive simulation and in the real world with a fleet of hand-sized quadcopters in controlled indoor environments with obstacles.