We investigate twisted double bilayer graphene (TDBG), a four-layer system composed of two AB-stacked graphene bilayers rotated with respect to each other by a small angle. Our ab initio band structure calculations reveal a considerable energy gap at the charge-neutrality point that we assign to the intrinsic symmetric polarization (ISP). We then introduce the ISP effect into the tight-binding parametrization and perform calculations on TDBG models that include lattice relaxation effects down to very small twist angles. We identify a narrow region around the magic angle characterized by a manifold of remarkably flat bands gapped out from other states even without external electric fields. To understand the fundamental origin of the magic angle in TDBG, we construct a continuum model that points to a hidden mathematical link to the twisted bilayer graphene model, thus indicating that the band flattening is a fundamental feature of TDBG and is not a result of external fields.