We infer conditional swap rate moments model independently from swaption cubes. Conditional volatility and skewness exhibit systematic variation across swap maturities and option expiries (conditional kurtosis less so), with conditional skewness sometimes changing sign. Conditional skewness displays some relation to the level and volatility of swap rates but is most consistently related to the conditional correlation between swap rates and swap rate variances. From realized excess returns on synthetic variance and skewness swap contracts, we infer that variance and (to a lesser extent) skewness risk premia are negative and time varying. For the most part, results hold true in both the USD and EUR markets and in both precrisis and crisis subsamples. We design and estimate a dynamic term structure model that captures much of the dynamics of conditional swap rate moments.