Spin qubits in silicon and germanium quantum dots are promising platforms for quantum computing, but entangling spin qubits over micrometer distances remains a critical challenge. Current prototypical architectures maximize transversal interactions between qubits and microwave resonators, where the spin state is flipped by nearly resonant photons. However, these interactions cause backaction on the qubit that yields unavoidable residual qubit-qubit couplings and significantly affects the gate fidelity. Strikingly, residual couplings vanish when spin-photon interactions are longitudinal and photons couple to the phase of the qubit. We show that large and tunable spin-photon interactions emerge naturally in state-of-the-art hole spin qubits and that they change from transversal to longitudinal depending on the magnetic field direction. We propose ways to electrically control and measure these interactions, as well as realistic protocols to implement fast high-fidelity two-qubit entangling gates. These protocols work also at high temperatures, paving the way toward the implementation of large-scale quantum processors.