Polarization of light has been widely used as a contrast mechanism in two-dimensional (2D) microscopy and also in some three-dimensional (3D) imaging modalities. In this paper, we report the 3D tomographic reconstruction of the refractive index (RI) tensor using 2D scattered fields measured for different illumination angles and polarizations. Conventional optical diffraction tomography (ODT) has been used as a quantitative, label-free 3D imaging method. It is based on the scalar formalism, which limits its application to isotropic samples. We achieve imaging of the birefringence of 3D objects through a reformulation of ODT based on vector diffraction theory. The off-diagonal components of the RI tensor reconstruction convey additional information that is not available in either conventional scalar ODT or 2D polarization microscopy. Finally, we show experimental reconstructions of 3D objects with a polarization-sensitive contrast metric quantitatively displaying the true birefringence of the samples. (C) 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement