Multiple lines of evidence at the individual and population level strongly suggest that infection hotspots, or superspreading events, where a single individual infects many others, play a key role in the transmission dynamics of COVID-19. However, most of the existing epidemiological models either assume or result in a Poisson distribution of the number of infections caused by a single infectious individual, often called secondary infections. As a result, these models overlook the observed overdispersion in the number of secondary infections and are unable to accurately characterize infection hotspots. In this work, we aim to fill this gap by introducing a temporal point process framework that explicitly represents sites where infection hotspots may occur. Under our model, overdispersion on the number of secondary infections emerges naturally. Moreover, using an efficient sampling algorithm, we demonstrate how to apply Bayesian optimization with longitudinal case data to estimate the transmission rate of infectious individuals at sites they visit and in their households, as well as the mobility reduction due to social distancing. Simulations using fine-grained demographic data and site locations from several cities and regions demonstrate that our framework faithfully characterizes the observed longitudinal trend of COVID-19 cases. In addition, the simulations show that our model can be used to estimate the effect of testing, contact tracing, and containment at an unprecedented spatiotemporal resolution, and reveal that these measures do not decrease overdispersion in the number of secondary infections.