Silicon nitride (SiN) micro-and nanomechanical resonators have attracted a lot of attention in various research fields due to their exceptionally high quality factors (Qs). Despite their popularity, the origin of the limiting loss mechanisms in these structures has remained controversial. In this Letter we propose an analytical model combining acoustic radiation loss with intrinsic loss. The model accurately predicts the resulting mode-dependent Qs of low-stress silicon-rich and high-stress stoichiometric SiN membranes. The large acoustic mismatch of the low-stress membrane to the substrate seems to minimize radiation loss and Qs of higher modes (n Lambda m >= 3) are limited by intrinsic losses. The study of these intrinsic losses in low-stress membranes reveals a linear dependence with the membrane thickness. This finding was confirmed by comparing the intrinsic dissipation of arbitrary (membranes, strings, and cantilevers) SiN resonators extracted from literature, suggesting surface loss as ubiquitous damping mechanism in thin SiN resonators with Q(surf) = beta h and beta = 6 x 10(10) +/- 4 x 10(10) m(-1). Based on the intrinsic loss the maximal achievable Qs and Qf products for SiN membranes and strings are outlined.