Breast cancer is the most common, and the number one cause of death by cancer among women. However, when it is sufficiently early detected, heavy treatments can be avoided, and morbidity and mortality can be reduced. This is the reason why randomized clinical trials were started in the 60s, followed in the last decades by screening mammography national programs. The process of breast cancer detection in mammography is complex. Its understanding offers numerous challenges to radiologists and medical physicists. One way to apprehend it is to model this process step-by-step by performing psychophysical experiments with anatomical or synthetic images. In this approach, the images information content is controlled. Since the first experiments with synthetic images created with white noise and simple geometric signals, technical and computational improvements allowed to get ever closer to clinical realism for studying the mechanism of perception of a signal on a radiological image. The present work extends the list of tools that have been used until now in psychophysical experiments in mammography. It proposes a detailed statistical analysis of anatomical images, from which algorithms for breast density classification and realistic breast texture synthesis are developed. In a second phase, psychophysical experiments with simple signals and benign or malignant masses combined with anatomical or synthetic backgrounds are presented. The performance of human observers is analyzed as a function of parameters such as background type, signal type, or uncertainty about the size or the shape of the signal. These results are compared to that of existing or adapted models from the literature, and the different models are evaluated in their ability to predict the performance of human observers for the detection of lesions in such conditions. Each step of this project focused on the objective and reproducible aspect of the image evaluation or of the observer performance. Controlled yet realistic conditions ensure the robustness of the results, as well as their clinical adaptability. Among the main results, synthetic mammographic texture images have been generated and validated. They provide a virtually unlimited database of images with demonstrated visual and statistical realism. Concerning the analysis of the human observers' performance, this work shows that they are sensitive to uncertainty about the signal size but not about its shape, that they use similar detection strategies with real and synthetic images, and that they are mainly sensitive to the anatomical fluctuations in the immediate area around the signal location. These effects, as well as the performance level of human observers for the detection tasks under consideration, could be reproduced by models taking into account human visual system characteristics. The conclusions of this work will be able to guide future studies in the field of detection tasks in mammography or tomosynthesis. This 3D breast imaging technique presents, like mammography, numerous challenges in order to understand and to objectively characterize its clinical potential. Studies with model observers specifically validated for detection tasks in medical imaging provide an excellent alternative in terms of time and costs for answering these questions.