Abstract: The automatic recognition of a wagon number plays an important role in railroad transportation systems. However, the wagon number character only occupies a very small area of the entire wagon image, and it is often accompanied by uneven illumination, a complex background, image contamination, and character stroke breakage, which makes the high-precision automatic recognition difficult. In recent years, object detection algorithm based on deep learning has made great progress, and it provides a solid technical basis for us to improve the performance of the train number recognition algorithm. This paper proposes a two-phase efficient wagon number recognition algorithm based on the high-performance YOLOv3 object detection algorithm. The entire recognition process is divided into two phases. In the first phase, the region of the wagon number in an image is detected from a low-resolution global image; in the second stage, the characters are detected in a high-resolution local image, formed into the wagon number according to their spatial position, and the final wagon number is obtained after verification based on the recognition confidence of each character and international wagon number coding rules. In addition, we proposed a new deep learning network-pruning algorithm based on the batch normalize scale factor and filter correlation. The importance of every filter was computed by considering the correlation between filter weights and the scale factor generated via batch normalization. By pruning and retraining the region detection model and character detection model, the storage space occupation and computational complexity were reduced without sacrificing recognition accuracy (which is even slightly improved in our experiment). Finally, we tested the proposed two-phase wagon number recognition algorithm on 1072 images from practical engineering application scenarios, and the results show that the proposed algorithm achieves 96.9% of the overall correct ratio (here, “correct” means all 12 characters are detected and recognized correctly), and the average recognition time is only 191 ms.