One of the problems of non-invasive Brain-Computer Interface (BCI) applications is the occurrence of anomalous (unexpected) signals that might degrade BCI performance. This situation might slip the operator’s attention since raw signals are not usually continuously visualized and monitored during BCI-actuated device operation. Anomalous data can for instance be the result of electrode misplacement, degrading impedance or loss of connectivity. Since this problem can develop at run-time, there is a need of a systematic approach to evaluate electrode reliability during online BCI operation. In this paper, we propose two metrics detecting how much each channel is deviating from its expected behavior. This quantifies electrode reliability at run-time which could be embedded into BCI data processing to increase performance. We manifest the effectiveness of these metrics in quantifying signal degradation by conducting three experiments: electrode swap, electrode manipulation and offline artificially degradation of P300 signals.