We study of the role of galaxy-galaxy interactions and disk instabilities in producing starburst activity in galaxies out to z = 4. For this, we use a sample of 387 galaxies with robust total star formation rate measurements from Herschel, gas masses from the Atacama Large Millimeter/submillimeter Array, stellar masses and redshifts from multiband photometry, and JWST/NIRCam rest-frame optical imaging. Using mass-controlled samples, we find an increased fraction of interacting galaxies in the starburst regime at all redshifts out to z = 4. This increase correlates with star formation efficiency (SFE) but not with gas fraction. However, the correlation is weak (and only significant out to z = 2), which could be explained by the short duration of SFE increase during interaction. In addition, we find that isolated disk galaxies make up a significant fraction of the starburst population. The fraction of such galaxies with star-forming clumps (“clumpy disks”) is significantly increased compared to the main-sequence disk population. Furthermore, this fraction directly correlates with SFE. This is direct observational evidence for a long-term increase of SFE maintained due to disk instabilities, contributing to the majority of starburst galaxies in our sample and hence to substantial mass growth in these systems. This result could also be of importance for explaining the growth of the most massive galaxies at z > 6.