In order to assist the field of neural networks in its maturing, a formalization and a solid foundation are essential. Additionally, to permit the introduction of formal proofs, it is essential to have an all encompassing formal mathematical definition of a neural network. Most neural networks, even biological ones, exhibit a layered structure. This publication shows that all neural networks can be represented as layered structures. This layeredness is therefore chosen as the basis for a formal neural network framework. This publication offers a neural network formalization consisting of a topological taxonomy, a uniform nomenclature, and an accompanying consistent mnemonic notation. Supported by this formalization, both a flexible hierarchical and a universal mathematical definition are presented.