Many algorithms have been proposed to solve the problem of matching feature points in two or more images using geometric assumptions to increase the robustness of the matching. However, these assumptions do not always hold; in particular, few methods address the problem of reliable matching in cases where it is unknown whether the images have any corresponding areas or objects in the first place. We propose two algorithms for matching feature points without the use of geometric constraints. The first relies on the fact that any match between two images should be better than all possible matches within a single image. The second algorithm extends this idea by using community structure in the similarity graph of feature points to find reliable correspondences. To evaluate the algorithms experimentally, we introduce a simple method to generate a large amount of test cases based on a set of image pairs with viewpoint changes. Our results show that the proposed algorithm is generally superior to traditional approaches in finding correct correspondences.