The context-based convolutional neural network (CNN) is one of the most well-known CNNs to improve the performance of semantic segmentation. It has achieved remarkable success in various medical image segmentation tasks. However, extracting rich and useful context information from complex and changeable medical images is a challenge for medical image segmentation. In this study, a novel Multi-Receptive-Field CNN (MRFNet) is proposed to tackle this challenge. MRFNet offers the optimal receptive field for each subnet in the encoder-decoder module (EDM) and generates multi-receptive-field context information at the feature map level. Moreover, MRFNet fuses these multi-feature maps by the concatenation operation. MRFNet is evaluated on 3 public medical image data sets, including SISS, 3DIRCADb, and SPES. Experimental results show that MRFNet achieves the outstanding performance on all 3 data sets, and outperforms other segmentation methods on 3DIRCADb test set without pre-training the model.