We introduce Neural Network (NN for short) approximation architectures for the numerical solution of Boundary Integral Equations (BIEs for short). We exemplify the proposed NN approach for the boundary reduction of the potential problem in two spatial dimensions. We adopt a Galerkin formulation-based method, in polygonal domains with a finite number of straight sides. Trial spaces used in the Galerkin discretization of the BIEs are built by using NNs that, in turn, employ the so-called Rectified Linear Units (ReLU) as the underlying activation function. The ReLU-NNs used to approximate the solutions to the BIEs depend nonlinearly on the parameters characterizing the NNs themselves. Consequently, the computation of a numerical solution to a BIE by means of ReLU-NNs boils down to a fine tuning of these parameters, in network training. We argue that ReLU-NNs of fixed depth and with a variable width allow us to recover well-known approximation rate results for the standard Galerkin Boundary Element Method (BEM). This observation hinges on existing well-known properties concerning the regularity of the solution of the BIEs on Lipschitz, polygonal boundaries, i.e. accounting for the effect of corner singularities, and the expressive power of ReLU-NNs over different classes of functions. We prove that shallow ReLU-NNs, i.e. networks having a fixed, moderate depth but with increasing width, can achieve optimal order algebraic convergence rates. We propose novel loss functions for NN training which are obtained using computable, local residual a posteriori error estimators with ReLU-NNs for the numerical approximation of BIEs. We find that weighted residual estimators, which are reliable without further assumptions on the quasi-uniformity of the underlying mesh, can be employed for the construction of computationally efficient loss functions for ReLU-NN training. The proposed framework allows us to leverage on state-of-the-art computational deep learning technologies such as TENSORFLOW and TPUs for the numerical solution of BIEs using ReLU-NNs. Exploratory numerical experiments validate our theoretical findings and indicate the viability of the proposed ReLU-NN Galerkin BEM approach.