(001)-textured AlN thin films as needed for bulk acoustic wave devices exhibit large mechanical stress variations as a function of growth substrate properties. We studied the relationship between stress and the surface morphology of a thermally oxidized silicon substrate that was modified by a thin amorphous silicon layer. A rms roughness of 0.1–1.1 nm of the latter resulted in an increase in mechanical stress in the subsequently sputtered AlN thin film going from −700 to +200 MPa. At the same time, the x-ray rocking curve width of AlN increased from 1.3° to 2.3°. The roughness of the Si interlayer was controlled by the Ar sputter pressure. Interestingly, the maximal roughness is obtained at an intermediate pressure. This is explained by an interplay of nucleation and diffusion phenomena governed by the kinetics of impinging atoms and ions. The Si interlayer was essential to avoid cracking of membranes exhibiting mixed Pt and SiO2 surfaces below the AlN film.