We study spaces of modelled distributions with singular behaviour near the boundary of a domain that, in the context of the theory of regularity structures, allow one to give robust solution theories for singular stochastic PDEs with boundary conditions. The calculus of modelled distributions established in Hairer (Invent Math 198(2):269-504, 2014. 10.1007/s00222-014-0505-4) is extended to this setting. We formulate and solve fixed point problems in these spaces with a class of kernels that is sufficiently large to cover in particular the Dirichlet and Neumann heat kernels. These results are then used to provide solution theories for the KPZ equation with Dirichlet and Neumann boundary conditions and for the 2D generalised parabolic Anderson model with Dirichlet boundary conditions. In the case of the KPZ equation with Neumann boundary conditions, we show that, depending on the class of mollifiers one considers, a boundary renormalisation takes place. In other words, there are situations in which a certain boundary condition is applied to an approximation to the KPZ equation, but the limiting process is the Hopf-Cole solution to the KPZ equation with a different boundary condition.