Context. Clumps in the rest-frame UV emission of galaxies at z less than or similar to 3 have been observed for decades. Since the launch of the James Webb Space Telescope (JWST), a large population is detected in the rest-frame near-infrared (NIR), raising questions about their formation mechanism. Aims. We investigate the presence and properties of NIR overdensities (hereafter substructures, including clumps) in star-forming and quiescent galaxies at 1 < z < 4 to understand their link to the evolution of their host galaxy. Methods. We identified substructures in JWST/NIRCam F277W and F444W residual images at a rest-frame wavelength of 1 mu m. Results. The fraction of galaxies with substructures with M-star > 109 M circle dot has steadily decreased with cosmic time from 40% at z = 4 to 10% at z = 1. NIR clumps, the most common type of small substructure, are much fainter (2% of the total galaxy flux) than similar UV clumps in the literature. Nearly all galaxies at the high-mass end of the main sequence (MS), starburst, and green valley regions have substructures. However, we do not find substructures in low-mass galaxies in the green valley and red sequence. Although massive galaxies on the MS and in the green valley have a 40% probability of hosting multiple clumps, the majority of clumpy galaxies host only a single clump. Conclusions. The fraction of clumpy galaxies in the rest-frame NIR is determined by the stellar mass and star formation rate (SFR) of the host galaxies. Its evolution with redshift is due to galaxies moving toward lower SFRs at z less than or similar to 2 and the buildup of low-mass galaxies in the green valley and red sequence. Based on their spatial distribution in edge-on galaxies, we infer that most substructures are produced in situ via disk fragmentation. Galaxy mergers may still play a role at high stellar masses, especially at a low SFR.