We consider the problem of sampling at unknown locations. We prove that, in this setting, if we take arbitrarily many samples of a polynomial or real bandlimited signal, it is possible to find another function in the same class, arbitrarily far away from the original, that could have generated the same samples. In other words, the error can be arbitrarily large.Motivated by this, we prove that, for polynomials, if the sample positions are constrained such that they can be described by an unknown rational function, uniqueness can be achieved.In addition to our theoretical results, we show that, in 1-D, the problem of recovering a painted surface from a single image exactly fits this framework. Furthermore, we propose a simple iterative algorithm for recovering both the surface and the texture and test it with simple simulations.