Prior modal stability analysis (Kojima et al., Phys. Fluids, vol. 27, 1984, pp. 19-32) predicted that a rising or sedimenting droplet in a viscous fluid is stable in the presence of surface tension no matter how small, in contrast to experimental and numerical results. By performing a non-modal stability analysis, we demonstrate the potential for transient growth of the interfacial energy of a rising droplet in the limit of inertialess Stokes equations. The predicted critical capillary numbers for transient growth agree well with those for unstable shape evolution of droplets found in the direct numerical simulations of Koh & Leal (Phys. Fluids, vol. 1, 1989, pp. 1309-1313). Boundary integral simulations are used to delineate the critical amplitude of the most destabilizing perturbations. The critical amplitude is negatively correlated with the linear optimal energy growth, implying that the transient growth is responsible for reducing the necessary perturbation amplitude required to escape the basin of attraction of the spherical solution.