Years of a fierce competition have naturally selected the fittest deep learning algorithms. Yet, although these models work well in practice, we still lack a proper characterization of why they do so. This poses serious questions about the robustness, trust, and fairness of modern AI systems. This thesis aims to contribute to bridge this gap by advancing the empirical and theoretical understanding of deep learning, with a specific emphasis on understanding the intricate relationship between weight space and function space and how this shapes the inductive bias. Our investigation starts with the simplest possible learning scenario: learning linearly separable hypotheses. Despite its simplicity, our analysis reveals that most networks have a nuanced inductive bias on these tasks that depends on the direction of separability. Specifically, we show that this bias can be encapsulated in an ordered sequence of vectors, the neural anisotropy directions (NADs), which encode the preference of the network to separate the training data in a given direction. The NADs can be obtained by randomly sampling the weight space. This not only establishes a strong connection between the functional landscape and the directional bias of each architecture but also offers a new lens for examining inductive biases in deep learning. We then turn our attention to modelling the inductive bias towards a more generalized set of hypotheses. To do so, we explore the applicability of the neural tangent kernel (NTK) as an analytical tool to approximate the functional landscape. Our research shows that NTK approximations can indeed gauge the relative learning complexities across numerous tasks, even when they cannot predict absolute network performance. This approximation works best when the learned weights lie close to the initialization. This provides a nuanced understanding of the NTK's ability in capturing inductive bias, laying the groundwork for its application in our subsequent investigations. The thesis then explores two critical issues in the deep learning research. First, we scrutinize implicit neural representations (INRs) and their ability to encode rich multimedia signals. Drawing inspirations on harmonic analysis and our earlier findings, we show that the NTK's eigenfunctions act as dictionary atoms whose inner product with the target signal determines the final reconstruction performance. INRs, which use sinusoidal embeddings to encode the input, can modulate the NTK so that its eigenfunctions constitute a meaningful basis. This insight has the potential to accelerate the development of principled algorithms in INRs, offering new avenues for architectural improvements and design. Second, we offer an extensive study of the conditions required for direct model editing in the weight space. Our analysis introduces the concept of weight disentanglement as the crucial factor enabling task-specific adjustments via task arithmetic. This property emerges during pre-training and is evident when distinct weight space directions govern separate, localized input regions of the function space. Significantly, we find that linearizing models by fine-tuning them in their tangent space enhances weight disentanglement, leading to performance improvements across edition benchmarks and models. In summary, our work unveils fresh insights into the fundamental links between weight space and function space, proposing a general framework for approximating inductive.