Abstract: Accurate recognition of colorectal images assists doctors in screening for malignant intestinal diseases. Colorectal cancer can be induced by colorectal polyps, ulcerative colitis, and papillary adenoma. Colorectal polyp segmentation can be used to rapidly locate polyps using automated colorectal polyp segmentation technology, saving the time and cost of manual screening and providing patients with valuable treatment time. Therefore, designing an automatic identification and accurate segmentation method is crucial for clinicians to improve diagnosis efficiency. Aiming at the problems in colorectal polyp image segmentation, such as large regional scale variation, variable location, blurred edges, and low contrast between polyps and normal tissues, which lead to low accuracy of lesion segmentation and artifacts on the segmentation boundary, an adaptive network based on the Swin Transformer and graphical reasoning is proposed. First, the Swin Transformer encoder is used to extract the global context information of the input image layer by layer, weaken the interference of background information, and analyze the salient characteristics of the lesion area on a multiscale. Second, a global and local feature interaction module is proposed to enhance the spatial perception ability of the network on complex lesions, highlight the key position information of the target to be segmented, and accurately locate the target. Third, a region-guided graph inference module is used to mine the higher-order dominant relationship between prior information in the way of graph cyclic recurrence to promote the transmission of information between graphs. Fourth, an edge constraint graph inference module oriented to edge details is designed to integrate edge details and improve the segmentation effect and segmentation precision. Experiments were performed on the CVC-ClinicDB, Kvasir, CVC-ColonDB, and ETIS datasets. The Dice coefficients were 0.939, 0.926, 0.810, and 0.788, respectively, and the average intersection ratios were 0.889, 0.879, 0.731, and 0.710, respectively. The mean absolute errors were 0.006, 0.017, 0.030, and 0.012, respectively. Compared with the SSformer method based on the transformer structure, the Dice coefficients are increased by 2.3%, 0.1%, 3.8%, and 2.1%, respectively, and the average crossover ratio is increased by 1.6%, 0.1%, 3.4%, and 1.2%, respectively. The overall segmentation performance of the algorithm test is better than that of the existing method. The simulation results show that the image segmentation accuracy of colorectal polyps with complex shapes and structures, low contrast, and blurred edges is high.