Affine rank minimization algorithms typically rely on calculating the gradient of a data error followed by a singular value decomposition at every iteration. Because these two steps are expensive, heuristics are often used for approximations that reduce computational burden. In this paper, we propose one recovery scheme that merges the two steps and show that it actually admits provable recovery guarantees while operating on space proportional to the degrees of freedom in the problem.