This paper develops a unified approach to the analysis and design of adaptive filters with error nonlinearities. In particular, the paper performs stability and steady-state analysis of this class of filters under weaker conditions than what is usually encountered in the literature, and without imposing any restriction on the color or statistics of the input. The analysis results are subsequently used to derive an expression for the optimum nonlinearity, which turns out to be a function of the probability density function of the estimation error. Some common nonlinearities are shown to be approximations to the optimum nonlinearity. The framework pursued here is based on energy conservation arguments.