The integration of renewable energy in modern power systems has increased the stochasticity of load patterns. Probabilistic load forecasting (PLF) acts as a solution to capture the uncertainty of load data. Especially, neural network (NN)-based models can be flexibly adapted for PLF. However, the black-box characteristic of NN makes it difficult for us to understand where the uncertainty comes from, thereby reducing the reliability of the forecast. This paper provides a model-agnostic NN explanation framework for PLF, attributing the uncertainty quantification of the probabilistic forecast to the uncertainty of the input feature to evaluate where the uncertainty in the forecast comes from. Unlike traditional deep learning explanation techniques that examine how the information of the feature affects the result, our explanation focuses on exploring how the uncertainty of the feature affects the result. In this way, we can know the uncertainty and error sources of PLF and further understand its mechanism, which can help reduce the adverse effects of data uncertainty on forecasts.