One of the main goals of Wendelstein 7-X (W7-X) is to demonstrate the fast particle confinement properties of the quasi-isodynamic stellarator concept. Fast particle populations will be produced either by Neutral Beam Injection (NBI) or by minority Ion Cyclotron Resonant Heating (ICRH). A fraction of these particles are expected to be lost (even without collisions), despite the optimisation procedure used for the W7-X design. Confinement properties of NBI particles in W7-X were presented in the paper of Drevlak et al (2014 Nucl. Fusion 54 073002). A detailed study is presented here where the loss patterns of an NBI population are described. In particular, focussing on a high-mirror equilibrium, the confinement of fast ions with varying energy injection is studied under collisional conditions. It is found that collisions are not only responsible for classical transport losses but also enhance drift induced losses caused by trapped particles. Moreover, an asymmetry is found in the toroidal position of particle losses which can be explained by local variation in the equilibrium field. The effects of a neoclassically resolved radial electric field are also investigated. Fast particle confinement is significantly improved by the associated ExB drift. In particular, an increasing radial electric field helps to reduce and even stop the losses due to the 3D equilibrium structure for times comparable to slowing down time.