Evaluating the visual quality of distorted images is a crucial task in the field of image compression, as artifacts may significantly impair the appeal and fidelity of images, therefore reducing the user experience. As assessing the visual quality of images through subjective visual quality experiments is often not feasible, objective image quality metrics are considered to be very attractive alternatives. Traditional objective image quality metrics, such as the Peak Signal-to-Noise Ratio and Structural Similarity Index, have long been used to assess compression artifacts. However, due to the complexity of human perception, estimated objective visual quality scores often diverge from their subjective counterparts. Recent advancements in deep learning have led to the development of learning-based metrics that promise to estimate the perceived visual quality of images with better accuracy. While learning-based methods have demonstrated enhanced performance compared to conventional methods on a number of datasets, their generalization performance across different quality ranges and artifacts has not been assessed yet. This paper presents a benchmarking study of conventional and learning-based objective image quality metrics while focusing solely on image compression artifacts. The experimental framework includes five source images with various contents compressed with five legacy and recent compression algorithms at four different quality levels, specifically focusing on the high-quality range. Results indicate that, in many cases, learning-based metrics present a higher correlation with human visual perception when compared to conventional methods, highlighting the potential of integrating such metrics in the development and refinement of image compression techniques.