Thermal position fluctuations of a colloidal particle in an optical trap are measured with microsecond resolution using back-focal-plane interferometry. The mean-square displacement MSD(t) and power spectral density are in excellent agreement with the theory for a Brownian particle in a harmonic potential that accounts for hydrodynamic memory effects. The motion of a particle is dominated at short times by memory effects and at longer times by the potential. We identify the time below which the particle's motion is not influenced by the potential, and find it to be approximately tau_k/20, where tau_k is the relaxation time of the restoring force of the potential. This allows us to exclude the existence of free diffusive motion (MSD(t) proprtional t) even for a sphere with a radius as small as 0.27 µm in a potential as weak as 1.5 µN/m. As the physics of Brownian motion can be used to calibrate an optical trap, we show that neglecting memory effects leads to an underestimation of more than 10% in the detector sensitivity and the trap stiffness for an experiment with a micrometer-sized particle and a sampling frequency above 200 kHz. Furthermore, these calibration errors increase in a nontrivial fashion with particle size, trap stiffness, and sampling frequency. Finally, we present a method to evaluate calibration errors caused by memory effects for typical optical trapping experiments.