One of the problems of non-invasive BCI applications is the noise and change of EEG recordings that can result in performance degradation. Signal noise and changes can for instance be the result of electrode misplacements, impedances degradation or complete loss of connectivity. Moreover, a more extended use of BCI systems outside research lab by lay operators necessitates the need of a systematic approach to evaluate signal reliability during online BCI operation to notify the operator to take a reaction. In this paper, we propose a method for quantifying how much each channel is deviating from its expected behavior. The proposed method is applied on a motor imagery dataset and the results show the correlation between the accuracy changes of the BCI and the feature deviation value.