Reconstructing lens potentials and lensed sources can easily become an underconstrained problem, even when the degrees of freedom are low, due to degeneracies, particularly when potential perturbations superimposed on a smooth lens are included. Regularization has traditionally been used to constrain the solutions where the data failed to do so, e.g. in unlensed parts of the source. In this exploratory work, we go beyond the usual choices of regularization and adopt observationally motivated priors for the source brightness. We also perform a similar comparison when reconstructing lens potential perturbations, which are assumed to be stationary, i.e. permeate the entire field of view. We find that physically motivated priors lead to lower residuals, avoid overfitting, and are decisively preferred within a Bayesian quantitative framework in all the examples considered. For the perturbations, choosing the wrong regularization can have a detrimental effect that even high-quality data cannot correct for, while using a purely smooth lens model can absorb them to a very high degree and lead to biased solutions. Finally, our new implementation of the semi-linear inversion technique provides the first quantitative framework for measuring degeneracies between the source and the potential perturbations.