In the theory of the bottom-up assembly of cosmic structures, one of the main challenges is to connect the smallest, most inconspicuous galaxies we observe today to the building blocks of more massive galaxies such as our own, the Milky Way. Do these so-called dwarf galaxies resemble the first galaxies formed in the Universe? Their small size and the preponderance of dark matter in dwarf galaxies, as well as their inner mass distribution, reinforce this idea. These characteristics make them excellent probes for cosmological models on small scales and the nature of dark matter. Having the simplest assembly histories of all galaxies, dwarfs are also dominated by old stellar populations. They underwent little to no further evolution after the Epoch of Reionization and constitute pristine relics of the early-Universe. In the first part of this thesis, using cosmological simulations, we show the impact of initial conditions generated in the early Universe, particularly primordial magnetic fields (PMFs), on the formation and evolution of dwarf galaxies. We show that the impact of primordial fields on galaxy formation is twofold: 1. The number of dark halos around the Milky Way able to host dwarf galaxies is boosted. 2. An earlier onset of star formation plus a higher rate of forming stars induces a large amount of ionizing photons at high redshifts. Through this work we show that PMFs have a significant role in initiating the structure formation in the Universe, and obtain strong constraints on the properties of PMFs. Then we address the impact of PMFs on galaxy formation in opposition to the pressure support produced by magnetic fields in the interstellar medium of galaxies, to find the dominant agent. We conclude that because of the shallow potential well in low mass dwarfs, magnetic pressure not only can immensely decrease the stellar mass, but also can delay the onset of star formation. This effect can entirely prevent the collapse of gas clouds in the lowest mass halos and result in an increased number of dark halos. Yet the impact of a primordially sourced magnetic field dominates over its later-time impacts. In the third part of this thesis, we target the elusive characteristics of the faintest galaxies, ultra faint dwarfs (UFDs). We explore how the first generations of metal-free stars (Pop III) could increase the mean metallicity of these galaxies. In this work, we show that massive Pop III stars which explode as energetic pair-instability supernovae increase the metal content of UFDs. However, being very rare and sometimes absent in the faintest UFDs, they have a limited impact on the global faint end of the metallicity-luminosity relation. Taken together these three parts have increased our knowledge of the early stages of galaxy formation. How the cosmology in the early-Universe, in particular the frequently neglected PMFs, impact the formation and evolution of the first galaxies. What is the impact of magnetic fields on the interstellar medium and the star formation history of dwarf galaxies? What is the impact of first stars on the metal-enrichment occurring during the process of galaxy formation? These questions have been explored in this PhD thesis.