Medical image segmentation is an important task forcomputer aided diagnosis. Pixelwise manual annotationsof large datasets require high expertise and is time consum-ing. Conventional data augmentations have limited benefitby not fully representing the underlying distribution of thetraining set, thus affecting model robustness when testedon images captured from different sources. Prior workleverages synthetic images for data augmentation ignor-ing the interleaved geometric relationship between differentanatomical labels. We propose improvements over previ-ous GAN-based medical image synthesis methods by jointlyencoding the intrinsic relationship of geometry and shape.Latent space variable sampling results in diverse generatedimages from a base image and improves robustness. Giventhose augmented images generated by our method, we trainthe segmentation network to enhance the segmentation per-formance of retinal optical coherence tomography (OCT)images. The proposed method outperforms state-of-the-art segmentation methods on the public RETOUCH datasethaving images captured from different acquisition proce-dures. Ablation studies and visual analysis also demon-strate benefits of integrating geometry and diversity.