This paper discusses the use of ferns (a set of binary features) for face detection. The binary feature used here is the sign of pixel intensity difference. Ferns were first introduced for keypoint recognition and showed good performance, and improving the speed of recognition. Keypoint recognition deals with classification of few hundred different classes, while face detection is a two-class problem with an unbalanced data. For keypoint recognition random pixel pairs proved to be good enough while we used conditional mutual information criteria to select a small subset of informative binary feature to build class conditional densities and a Naive Bayesian classifier is used for face and non-face classification. We compared our approach with boosted haar-like features, modified census transform (MCT,',','), and local binary pattern on a single stage classifier. Results shows that ferns when compared to haar-like features are robust to illumination changes and comparable to boosted MCT feature. Finally a cascade of classifiers was built and the performance on cropped face images and the localization results using Jesorsky measure are reported on XM2VTS and BANCA database.