Precise positioning of single site-controlled inverted pyramidal InGaAs quantum dots (QDs) at the antinode of a GaAs photonic crystal cavity with nanometer-scale accuracy holds unique advantages compared to self-assembled QDs and offers great promise for practical on-chip photonic quantum information processing. However, the strong coupling regime in this geometry has not yet been achieved due to the low cavity Q-factor based on the (111)B-oriented membrane structures. Here, we reveal the onset of phonon-mediated coherent exciton-photon interaction on our tailored single site-controlled InGaAs QD-photonic crystal cavity. Our results present the Rabi-like oscillation of luminescence intensity between excitonic and photonic components correlated with their energy splitting pronounced at small detuning. Such Rabi-like oscillation is well reproduced by modeling the coherent exchange of the exciton-photon population. The modeling further reveals an oscillatory two-time covariance at QD-cavity resonance, which indicates that the system operates at the onset of the strong coupling regime. Moreover, by using the cavity mode as a probe of the virtual state of the QD induced by phonon scattering, it reveals an increase in phonon scattering rates near the QD-cavity resonance and asymmetric phonon emission and absorption rate of even around 50K.