We prove sharp pointwise $t^{-3}$ decay for scalar linear perturbations of a Schwarzschild black hole without symmetry assumptions on the data. We also consider electromagnetic and gravitational perturbations for which we obtain decay rates $t^{-4}$, and $t^{-6}$, respectively. We proceed by decomposition into angular momentum $\ell$ and summation of the decay estimates on the Regge-Wheeler equation for fixed $\ell$. We encounter a dichotomy: the decay law in time is entirely determined by the asymptotic behavior of the Regge-Wheeler potential in the far field, whereas the growth of the constants in $\ell$ is determined entirely by the behavior of the Regge-Wheeler potential in a small neighborhood around the photon sphere. In other words, the tails are controlled by small energies, whereas the number of angular derivatives needed on the data is determined by energies close to the top of the Regge-Wheeler potential. This dichotomy corresponds to the well-known principle that for initial times the decay reflects the presence of complex resonances generated by the potential maximum, whereas for later times the tails are determined by the far field. However, we do not invoke complex resonances at all, but rely instead on semiclassical Sigal-Soffer type propagation estimates based on a Mourre bound near the top energy.