Precisely localizing anomalies in medical images remains a significant challenge due to their heterogeneous nature across modalities and organs. While initial efforts excelled in identifying prominent anomalies, detecting minute target lesions posed significant limitations. These minute anomalies are particularly elusive and demand advanced detection techniques. Additionally, many existing models demand high computational resources, limiting their practicality in real-world clinical settings. In this study, we present REUnet, a novel Unet based architecture designed to address these obstacles by providing precise segmentation while also exhibiting strong generalization across diverse modalities and organs. The core advantage of REUnet resides in its resilient encoding pathway, constructed upon a module called dynamic mobile inverted bottleneck convolution. This module introduces a gating signal that significantly enhances semantic information, enabling the model to focus on specific regions of interest. The encoding pathway of REUnet is also linked strategically with the decoder to ensure efficient processing of these robust features which facilitates better communication between the two. Furthermore, the use of depth-wise separable convolution and dropout layers further makes REUnet computationally efficient for clinical use. Extensive experiments conducted on five publicly available datasets, including DUKE, BRATS2020, KiTS2023, INBreast, and FracAtlas, demonstrate REUnet's strong generalization capabilities and superior performance, establishing a new state-of-the-art in medical image segmentation. The source code is available at GitHub link: https://github.com/labsroy007/RobustSemanticLearning.