Decoherence of a charge qubit is usually credited to charge noise in the environment. Here we show that charge noise may not be the limiting factor for the qubit coherence. To this end, we study coherence properties of a crystal-phase defined semiconductor nanowire double quantum dot (DQD) charge qubit strongly coupled to a high-impedance resonator using radio-frequency reflectometry. Response of this hybrid system is measured both at a charge noise sensitive operation point (with finite DQD detuning) and at an insensitive point (so-called sweet spot with zero detuning). A theoretical model based on the Jaynes-Cummings Hamiltonian matches the experimental results well and yields only a 10% difference in decoherence rates between the two cases, despite that the sensitivity to detuning charge noise differs by a factor of 5. Therefore, the charge noise is not limiting the coherence in this experiment with this type of semiconducting nanowire qubits.