In this paper, kinetic and potential energy transfers around a spectral test filter scale in direct numerical simulations of decaying stratified turbulence are studied in both physical and spectral domains. It is shown that while the domain-averaged effective subgrid scale energy transfer in physical space is a net downscale cascade, it is actually a combination of large values of downscale and upscale transfer, i.e. forward- and backscatter, in which the forward scatter is slightly dominant. Our results suggest that spectral backscatter in stratified turbulence depends on the buoyancy Reynolds number R-eb and the filtering scale Delta(test). When the test filter scale Delta(test) is around the dissipation scale L-d, transfer spectra show spectral backscatter from sub-filter to intermediate scales, as reported elsewhere. However, we find that this spectral backscatter is due to viscous effects at vertical scales around the test filter. It is also shown that there is a non-local energy transfer from scales larger than the buoyancy scale L-b to small scales. The effective turbulent Prandtl number spectra demonstrate that the assumption Pr-t approximate to 1 is reasonable for the local energy transfer. (C) 2016 Elsevier Masson SAS. All rights reserved.