Using site-controlled semiconductor quantum dots (QDs) free of multiexcitonic continuum states, integrated with photonic crystal membrane cavities, we clarify the effects of pure dephasing and phonon scattering on exciton-cavity coupling in the weak-coupling regime. In particular, the observed QD-cavity copolarization and cavity mode feeding versus QD-cavity detuning are explained quantitatively by a model of a two-level system embedded in a solid-state environment.