Owing to stochastic fluctuations arising from finite population size, known as genetic drift, the ability of a population to explore a rugged fitness landscape depends on its size. In the weak mutation regime, while the mean steady-state fitness increases with population size, we find that the height of the first fitness peak encountered when starting from a random genotype displays various behaviours versus population size, even among small and simple rugged landscapes. We show that the accessibility of the different fitness peaks is key to determining whether this height overall increases or decreases with population size. Furthermore, there is often a finite population size that maximizes the height of the first fitness peak encountered when starting from a random genotype. This holds across various classes of model rugged landscapes with sparse peaks, and in some experimental and experimentally inspired ones. Thus, early adaptation in rugged fitness landscapes can be more efficient and predictable for relatively small population sizes than in the large-size limit.This article is part of the theme issue 'Interdisciplinary approaches to predicting evolutionary biology'.