We investigate the error made by the pairwise summation (PWS) approximation in three geometries where the exact formula for the Casimir interaction is known: atom-slab, slab-slab, and sphere-slab configurations. For each case, the interactions are calculated analytically by summing the van der Waals interactions between the two objects. We show that the PWS result is incorrect even for an infinitely thin slab in the atom-slab configuration, because of local field effects, unless the material is infinitely dilute. In the experimentally relevant case of dielectric materials, in all considered geometries the error made by the PWS approximation is much higher than the well-known value obtained for perfect reflectors in the long-range regime. This error is maximized for permittivities close to the one of silicon. DOI: 10.1103/PhysRevB.87.045413