The structural incommensurate modulation (SIM) rule in hexagonal Ba(Ti1-xMx)O3-delta (1/18 <= x <= 1/3, M = Mn, Fe) multiferroic ceramics is investigated using transmission electron microscope (TEM). Our systematic analysis on the modulation characteristics in the electron diffraction patterns, especially along [001] direction, reveals that three types of anti-phase domain boundaries (APBs) dominate the SIMs. According to the derived APB models, the calculated oxygen vacancies at both O1 (face sharing) and O2 (corner sharing) sites qualitatively agree quite well with our previous results. Analysis on the high-resolution TEM images reveals that complex dislocation structures also play important roles in the formation of the SIMs. Copyright 2012 Author(s). This article is distributed under a Creative Commons Attribution 3.0 Unported License. [http://dx.doi.org/10.1063/1.4766181]