Abstract: Not only is scrap steel an indispensable ferritic raw material for the modern steel industry, but it is also the only green raw material that can replace iron ore in large quantities. The quality of the scrap steel directly affects the quality of molten steel, which makes it necessary to sort and grade scrap steel before it enters the furnace. Most iron and steel enterprises determine the grade of scrap steel mainly by visual inspection and caliper-based measurements by quality management personnel. As a result, this process is prone to human errors and low efficiency. Therefore, given that the major challenges of scrap inspection include the many categories of scrap, complex actual detection scenarios, and challenges in manual system connection, a deep learning network model CSSNet was proposed for scrap classification and rating based on the Squeeze-Excitation (SE) attention mechanism, and images of the scrap unloading process were collected for model training and validation. First, a 1∶3 physical model of scrap steel quality inspection was built to simulate this process. High-resolution visual sensors were used to collect images of diverse types of scrap steel in the scene of trucks unloading scrap steel. Then, a cross-stage local network was used to extract the features of the collected scrap images, the spatial pyramid structure was used to solve the problem of feature loss, and the attention mechanism was used to focus on the correlation between channels and retain the channel with the most feature information. Finally, model training and validation were done using two datasets containing seven labels for classification. In the model prediction stage, the constructed scrap steel quality inspection model CSSNet was used to judge the scrap steel category and quality to verify the accuracy and detection efficiency of the model. Based on the self-made scrap steel validation dataset, its performance was compared with mainstream single-stage object detection packages such as YOLOv4, YOLOv5s, and the two-stage object detection model Faster R-CNN. The model was found to be able to effectively rate different grades of scrap steel, with the classification accuracy rate of all categories has reached 83.7% and an mAP value of 88.8%. The performance of the CSSNet model is better than the other three target detection models. CSSNet can not only fully meet the needs of the actual production applications in terms of accuracy, real-time performance, and identification and rating efficiency but also surpass the traditional manual scrap quality inspection method, address multiple issues in the evaluation of scrap steel quality, and realize automated scrap steel quality testing.