Designing computer-assisted diagnosis (CAD) systems that can precisely identify lesions from mammography images would be useful for clinicians. Considering the morphological variation in breast cancer, it is necessary to extract robust features from the mammogram. Here, we propose a mass detection CAD system that is based on Faster R-CNN. First, we applied a novel convolution network in the backbone of Faster R-CNN, namely deformable convolution network (DCN), which improves the detection of lesions with varying shapes and sizes. Second, the original Faster R-CNN uses the output of the last layer of the backbone as a single-scale feature map. To facilitate the detection of small lesions, we used a multiscale feature pyramid network of multiple cross-scale connections between the different output layers of the backbone, called the neural architecture search-feature pyramid network (NAS-FPN). Thus, we were able to integrate the best features into the model. We then evaluated our method by using the datasets the Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM) and INbreast, respectively. Our method yielded a true positive rate of 0.9345 at 2.2805 false positive per image on CBIS-DDSM and a true positive rate of 0.9554 at 0.3829 false positive per image on INbreast. Graphical abstract.
Keywords: Deformable convolution network; Faster R-CNN; Mammography; Mass detection; Multiscale features.