In the field of mobile payment, due to the practical application scenarios and low fault tolerance requirements, it is still a difficult task to obtain card number information from the bank card image and to quickly and accurately identify the bank card number. On the basis of studying the target detection algorithm YOLOv3, this paper puts forward a bank card number detection algorithm which improves the network structure of YOLOv3. In the network prediction section, the branch of feature scale prediction is added to realize the calculation of the fourth layer feature scale. Then, before calculating each feature scale, the idea of spatial pyramid pooling is introduced to increase the capability of network feature fusion and improve the accuracy of network detection. Further, combining the improved YOLOv3 detection model with DenseNet, a card number recognition model for bank cards is given, and an optimized DenseNet card number recognition model is achieved after expanding the dataset with the combination of traditional image enhancement methods and model training. Finally, the experimental results showing that the method in this paper improves the average accuracy of the algorithm and model parameter control, which can better apply to the needs of card number recognition in practical application scenarios.