The technique of photometric redshifts has become essential for the exploitation of multi-band extragalactic surveys. While the requirements on photometric redshifts for the study of galaxy evolution mostly pertain to the precision and to the fraction of outliers, the most stringent requirement in their use in cosmology is on the accuracy, with a level of bias at the sub-percent level for the Euclid cosmology mission. A separate, and challenging, calibration process is needed to control the bias at this level of accuracy. The bias in photometric redshifts has several distinct origins that may not always be easily overcome. We identify here one source of bias linked to the spatial or time variability of the passbands used to determine the photometric colours of galaxies. We first quantified the effect as observed on several well-known photometric cameras, and found in particular that, due to the properties of optical filters, the redshifts of off-axis sources are usually overestimated. We show using simple simulations that the detailed and complex changes in the shape can be mostly ignored and that it is sufficient to know the mean wavelength of the passbands of each photometric observation to correct almost exactly for this bias; the key point is that this mean wavelength is independent of the spectral energy distribution of the source. We use this property to propose a correction that can be computationally efficiently implemented in some photometric-redshift algorithms, in particular template-fitting. We verified that our algorithm, implemented in the new photometric-redshift code Phosphoros, can effectively reduce the bias in photometric redshifts on real data using the CFHTLS T007 survey, with an average measured bias Delta z over the redshift range 0.4 <= z <= 0.7 decreasing by about 0.02, specifically from Delta z similar or equal to 0.04 to Delta z similar or equal to 0.02 around z = 0.5. Our algorithm is also able to produce corrected photometry for other applications.