The thermal position fluctuations of a single micron-sized sphere immersed in a fluid were recorded by optical trapping interferometry with nanometer spatial and microsecond temporal resolution. We find, in accord with the theory of Brownian motion including hydrodynamic memory effects, that the transition from ballistic to diffusive motion is delayed to significantly longer times than predicted by the standard Langevin equation. This delay is a consequence of the inertia of the fluid. On the shortest time scales investigated, the sphere's inertia has a small, but measurable, effect.