The performance of many color science and imaging algorithms are evaluated based on their mean errors. However, if these errors are not normally distributed, statistical evaluations based on the mean are not appropriate performance metrics. We present a non-parametric method, called the Wilcoxon signed-rank test, which can be used to evaluate performance without making any underlying assumption of the error distribution. When applying the metric to the performance of chromatic adaptation transforms on corresponding color data, we can derive a new CAT that statistically significantly outperforms CAT02 at the 95% confidence level.