Background Universal access to basic sanitation remains a global challenge, particularly in low- and middle-income countries. Efforts are underway to improve access to sanitation in informal settlements, often through shared facilities. However, access to these facilities and their potential health gains—notably, the prevention of diarrheal diseases—may be hampered by contextual aspects related to the physical environment. This study explored associations between the built environment and perceived safety to access toilets, and associations between the latter and diarrheal infections. Methods A cross-sectional study was carried out between July 2021 and February 2022, including 1714 households in two informal settlements in Abidjan (Côte d’Ivoire) and two in Nairobi (Kenya). We employed adjusted odds ratios (aORs) obtained from multiple logistic regressions (MLRs) to test whether the location of the most frequently used toilet was associated with a perceived lack of safety to use the facility at any time, and whether this perceived insecurity was associated with a higher risk of diarrhea. The MLRs included several exposure and control variables, being stratified by city and age groups. We employed bivariate logistic regressions to test whether the perceived insecurity was associated with settlement morphology indicators derived from the built environment. Results Using a toilet outside the premises was associated with a perceived insecurity both in Abidjan [aOR = 3.14, 95% confidence interval (CI): 1.13–8.70] and in Nairobi (aOR = 57.97, 95% CI: 35.93–93.53). Perceived insecurity to access toilets was associated with diarrheal infections in the general population (aOR = 1.90, 95% CI: 1.29–2.79 in Abidjan, aOR = 1.69, 95% CI: 1.22–2.34 in Nairobi), but not in children below the age of 5 years. Several settlement morphology features were associated with perceived insecurity, namely, buildings’ compactness, the proportion of occupied land, and angular deviation between neighboring structures. Conclusions Toilet location was a critical determinant of perceived security, and hence, must be adequately addressed when building new facilities. The sole availability of facilities may be insufficient to prevent diarrheal infections. People must also be safe to use them. Further attention should be directed toward how the built environment affects safety.