Regarding the surface structure of shaft gear parts is complex, the difficulties of detection are that complex background and reflection of shaft gear parts might interfere the detection. Besides, the different types of defects in different areas cannot be detected independently. Therefore, it is necessary to extract each machining region. Traditional region extraction methods cannot effectively separate the object area from the background area due to the complex background and unclear boundaries. A region extraction method for shaft gear parts was offered in this study, which was based on Mask R-CNN. First of all, ResNet50 was used as the feature extraction network to extract the features of the image; then the feature pyramid network (FPN) was applied to fuse the information of different scale feature maps; in the next step, the region of interest (ROI) was generated by the region proposal network (RPN); finally, the machining region was segmented by the head network. Combined with the structural characteristics of the shaft gear, the model parameters were constantly adjusted to improve the accuracy of the model training process. The trained model was tested on the manually marked shaft gear data set. AP (Average Precision) value was 89.2%. With regard to segmenting complex background regions, experimental data suggested that the Mask R-CNN method is an effective approach which could achieve high-precision extraction of the machining area for shaft gear parts.