We report small-angle neutron scattering measurements of the vortex lattice (VL) structure in single crystals of the lightly underdoped cuprate superconductor YBa2Cu3O6.85. At 2 K, and for fields of up to 16 T applied parallel to the crystal c-axis, we observe a sequence of field-driven and first-order transitions between different VL structures. By rotating the field away from the c-axis, we observe each structure transition to shift to either higher or lower field dependent on whether the field is rotated towards the [100] or [010] direction. We use this latter observation to argue that the Fermi surface morphology must play a key role in the mechanisms that drive the VL structure transitions. Furthermore, we show this interpretation is compatible with analogous results obtained previously on lightly overdoped YBa2Cu3O7. In that material, it has long-been suggested that the high field VL structure transition is driven by the nodal gap anisotropy. In contrast, the results and discussion presented here bring into question the role, if any, of a nodal gap anisotropy on the VL structure transitions in both YBa2Cu3O6.85 and YBa2Cu3O7.