The motion of an optically trapped sphere constrained by the vicinity of a wall is investigated at times where hydrodynamic memory is significant. First, we quantify, in bulk, the influence of confinement arising from the trapping potential on the sphere's velocity autocorrelation function C(t). Next, we study the splitting of C(t) into C-parallel to(t) and C-perpendicular to(t), when the sphere is approached towards a surface. Thereby, we monitor the crossover from a slow t(-3/2) long-time tail, away from the wall, to a faster t(-5/2) decay, due to the subtle interplay between hydrodynamic backflow and wall effects. Finally, we discuss the resulting asymmetric time-dependent diffusion coefficients.