Intelligent behaviour can emerge from the interaction of many simple units. Modern artificial neural networks are able to produce some form of it, but the training procedure we use to tune the interactions between these units tells us little about how it does so. We cannot say in advance whether a specific setup allows training to be successful, nor do we have a clear understanding of the solution it eventually finds. This thesis studies these questions through the lens of the loss landscape, the space of all possible configurations of a network's parameters and their associated error. The critical points of this landscape, the points at which the error gradient vanishes, are the main objects of study. Their arrangement changes as the network grows, and it is through this change that the two questions are approached: what adding neurons does to these points, and how the solutions reached at these points are structured and what computation they implement. First, we gather existing theoretical results on the geometry around critical points under a single mechanism, which we call Escape Dimensions: the minima of a smaller network persist as neurons are added, but the new dimensions of the larger landscape introduce escape routes, transforming these minima into escapable saddles. Globally, as this process repeats, escape dimensions accumulate and poor minima become increasingly rare relative to good ones, so that despite the non-convexity of the loss, training becomes more likely to succeed in the larger the network. We present this intuitive account in contrast to another one common in the field, the lottery ticket metaphor, whose assumptions we show do not hold in practice. We argue that the relative number of such saddles and of redundant solutions is what could bring a quantitative account about the trainability of neural networks, potentially predicting how large a network needs to be for training to succeed. We then show that this mechanism can be exploited: student networks given roughly four times as many neurons as an unknown teacher network can be reliably trained to imitate the teacher's function to zero error; in accordance with the quantitative predictions of the previous part. By characterising the degeneracy of these zero-error solutions, we find that the student's neurons copy the teacher's up to known symmetries, which permits the teacher's weights to be recovered from its input-output behaviour alone. Turning to the structure of solutions, we characterise one that arises from the transformation of a minimum into a saddle: flat, channel-like structures in the landscape running to infinite parameter norm. We describe them in terms of their asymptotic learning dynamics, the geometry of the landscape around them, and the function they implement. These results suggest that the critical points generated by a network's symmetries, and the way they are transformed as neurons are added, offer a useful way of approaching both the trainability of neural networks and the structure of the solutions they find.