Thanks to its high refractive index contrast, band gap, and polarization mismatch compared to GaN, In0.17Al0.83N layers lattice-matched to GaN are an attractive solution for applications such as distributed Bragg reflectors, ultraviolet light-emitting diodes, or high electron mobility transistors. In order to study the structural degradation mechanism of InAlN layers with increasing thickness, we performed metalorganic vapor phase epitaxy of InAlN layers of thicknesses ranging from 2 to 500 nm, on free-standing (0001) GaN substrates with a low density of threading dislocations, for In compositions of 13.5% (layers under tensile strain), and 19.7% (layers under compressive strain). In both cases, a surface morphology with hillocks is initially observed, followed by the appearance of V-defects. We propose that those hillocks arise due to kinetic roughening, and that V-defects subsequently appear beyond a critical hillock size. It is seen that the critical thickness for the appearance of V-defects increases together with the surface diffusion length either by increasing the temperature or the In flux because of a surfactant effect. In thick InAlN layers, a better (worse) In incorporation occurring on the concave (convex) shape surfaces of the V-defects is observed leading to a top phase-separated InAlN layer lying on the initial homogeneous InAlN layer after V-defects coalescence. It is suggested that similar mechanisms could be responsible for the degradation of thick InGaN layers. (C) 2013 American Institute of Physics. [http://dx.doi.org/10.1063/1.4790424]