In this paper we present a new method to enhance object detection by removing false alarms and merging multiple detections in a principled way with few parameters. The method models the output of an object classiï¬er which we consider as the context. A hierarchical model is built using the detection distribution around a target sub-window to discriminate between false alarms and true detections. Next the context is used to iteratively reï¬ne the detections. Finally the detections are clustered using the Adaptive Mean Shift algorithm. The speciï¬c case of face detection is chosen for this work as it is a mature ï¬eld of research. We report results that are better than baseline method on XM2VTS, BANCA and MIT+CMU face databases. We signiï¬cantly reduce the number of false acceptances while keeping the detection rate at approximately the same level and in certain conditions we recover miss-aligned detections.