The zero-temperature phase diagram of the J1-J2 SU(N) antiferromagnetic Heisenberg spin chain is investigated by means of complementary field theory and numerical approaches for general N. A fully gapped SU(N) valence bond solid made of N sites is formed above a critical value of J2/J1 for all N. We find that the extension of this N-merized phase for larger values of J2 strongly depends on the parity of N. For even N, the phase smoothly interpolates to the large J2 regime where the model can be viewed as a zigzag SU(N) two-leg spin ladder. The phase exhibits both a N-merized ground state and incommensurate spin-spin correlations. In stark contrast to the even case, we show that the N-merized phase with odd N has only a finite extent with no incommensuration. A gapless phase in the SU(N )1 universality class is stabilized for larger J2 that stems from the existence of a massless renormalization group flow from SU(N )2 to SU(N )1 conformal field theories when N is odd.