This paper provides a time-domain feedback analysis of gradient-based adaptive schemes with emphasis on stability and robustness issues. It is shown that an intrinsic feedback structure, mapping the noise sequence and the initial weight guess to the a priori estimation errors and the final weight estimate, can be associated with such schemes. The feedback configuration is motivated via energy arguments and is shown to consist of two major blocks: a time-variant lossless (i.e., energy preserving) feedforward path and a time-variant feedback path. The configuration is further shown to lend itself rather immediately to analysis via a so-called small gain theorem; thus leading to stability conditions that require the contractivity of certain operators.