Feature description for local image patch is widely used in computer vision. While the conventional way to design local descriptor is based on expert experience and knowledge, learning based methods for designing local descriptor become more and more popular because of their good performance and data-driven property. This paper proposes a novel data-driven method for designing binary feature descriptor, which we call Receptive Fields Descriptor (RFD). Technically, RFD is constructed by thresholding responses of a set of receptive fields, which are selected from a large number of candidates according to their distinctiveness and correlations in a greedy way. By using two different kinds of receptive fields (namely Rectangular pooling area and Gaussian pooling area) for selection, we obtain two binary descriptors RFDR and RFDG accordingly. Image matching experiments on the well known Patch Dataset and Oxford Dataset demonstrate that RFD significantly outperforms the state-of-the-art binary descriptors, and is comparable to the best float-valued descriptors at a fraction of processing time. Finally, experiments on object recognition tasks confirm that both RFDR and RFDG successfully bridge the performance gap between binary descriptors and their floating-point competitors.