Diffusion Magnetic Resonance Imaging (dMRI) is a powerful non-invasive method for studying white matter tracts of the brain. However, accurate microstructure estimation with fiber orientation distribution (FOD) using existing computational methods requires a large number of diffusion measurements. In clinical settings, this is often not possible for neonates and fetuses because of increased acquisition times and subject movements. Therefore, methods that can estimate the FOD from reduced measurements are of high practical utility. Here, we exploited deep learning and trained a neural network to directly map dMRI data acquired with as low as six diffusion directions to FODs for neonates and fetuses. We trained the method using target FODs generated from densely-sampled multiple-shell data with the multi-shell multi-tissue constrained spherical deconvolution (MSMT-CSD). Detailed evaluations on independent newborns’ test data show that our method achieved estimation accuracy levels on par with the state-of-the-art methods while reducing the number of required measurements by more than an order of magnitude. Qualitative assessments on two out-of-distribution clinical datasets of fetuses and newborns show the consistency of the estimated FODs and hence the cross-site generalizability of the method.