We have investigated the motion of a single optically trapped colloidal particle close to a limiting wall at time scales where the inertia of the surrounding fluid plays a significant role. The velocity autocorrelation function exhibits a complex interplay due to the momentum relaxation of the particle, the vortex diffusion in the fluid, the obstruction of flow close to the interface, and the harmonic restoring forces due to the optical trap. We show that already a weak trapping force has a significant impact on the velocity autocorrelation function C(t)=< v(t)v(0)> at times where the hydrodynamic memory leads to an algebraic decay. The long-time behavior for the motion parallel and perpendicular to the wall is derived analytically and compared to numerical results. Then, we discuss the power spectral densities of the displacement and provide simple interpolation formulas. The theoretical predictions are finally compared to recent experimental observations.