The orbits of the least chemically enriched stars open a window on the formation of our Galaxy when it was still in its infancy. The common picture is that these low-metallicity stars are distributed as an isotropic, pressure-supported component since these stars were either accreted from the early building blocks of the assembling Milky Way (MW), or were later brought by the accretion of faint dwarf galaxies. Combining the metallicities and radial velocities from the Pristine and LAMOST surveys and Gaia DR2 parallaxes and proper motions for an unprecedented large and unbiased sample of 1027 very metal poor stars at [Fe/H] <= -2.5 dex, we show that this picture is incomplete. We find that 31 per cent of the stars that currently reside spatially in the disc (vertical bar Z vertical bar <= 3 kpc) do not venture outside of the disc plane throughout their orbit. Moreover, this sample shows strong statistical evidence (at the 5.0 sigma level) of asymmetry in their kinematics, favouring prograde motion. The discovery of this population implies that a significant fraction of stars with iron abundances [Fe/H] <= -2.5 dex merged into, formed within, or formed concurrently with the MW disc and that the history of the disc was quiet enough to allow them to retain their disc-like orbital properties, challenging theoretical and cosmological models.