Semantic segmentation is an exciting research topic in medical image analysis because it aims to detect objects in medical images. In recent years, approaches based on deep learning have shown a more reliable performance than traditional approaches in medical image segmentation. The U-Net network is one of the most successful end-to-end convolutional neural networks (CNNs) presented for medical image segmentation. This paper proposes a multiscale Residual Dilated convolution neural network (MSRD-UNet) based on U-Net. MSRD-UNet replaced the traditional convolution block with a novel deeper block that fuses multi-layer features using dilated and residual convolution. In addition, the squeeze and execution attention mechanism (SE) and the skip connections are redesigned to give a more reliable fusion of features. MSRD-UNet allows aggregation of contextual information, and the network goes without needing to increase the number of parameters or required floating-point operations (FLOPS). The proposed model was evaluated on three multimodal datasets: polyp, skin lesion, and nuclei segmentation. The obtained results proved that the MSDR-Unet model outperforms several state-of-the-art U-Net-based methods.