Polarimetric imaging techniques demonstrate enhanced capabilities in advanced object detection tasks with their capability to discriminate man-made objects from natural background surfaces. While spectral signatures carry information only about material properties, the polarization state of an optical field contains information related to surface features of objects, such as, shape and roughness. With these additional benefits, polarimetric imaging reveal physical properties operable for advanced object detection tasks which are not possible to acquire by using conventional imaging. In this work, the primary objective is to utilize the state-of-the-art deep learning models designed for object detection tasks using images obtained by polarimetric systems. In order to train deep learning models, it is necessary to have a sufficiently large dataset consisting of polarimetric images with various classes of objects in them. We started by constructing such dataset with adequate number of visual and infrared (SWIR) polarimetric images obtained using polarimetric imaging systems and masking relevant parts for object detection models. We managed to achieve a high performance score while detecting vehicles with metallic surfaces using polarimetric imaging. Even with limited number of training samples, polarimetric imaging demonstrated superior performance comparing to models trained using conventional imaging techniques. We observed that using models trained with both polarimetric and conventional imaging techniques in parallel gives the best performance score since these models were able to compensate for each other's lacking points. In the subsequent stages, we plan to expand the study to the application of spiking neural network (SNN) architectures for implementing the detection/classification tasks.