Background Rectal cancer segmentation in CT is crucial for timely diagnosis. Despite promising methods, challenges remain due to the rectum’s complex anatomy and the lack of a comprehensive annotated dataset. Methods A total of 33,024 slice pairs from 398 rectal cancer patients in a new source center are enrolled into our dataset, named CARE Dataset, with pixel-level annotations for both normal and cancerous rectum tissue. We split it into 317 cases for training and 81 for testing. Additionally, we introduce a segmentation model, U-SAM, which, to the best of our knowledge, is a novel approach designed to handle the complex anatomy of the rectum by incorporating prompt information. Segmentation performance for both normal and cancerous rectum was evaluated using Intersection-over-Union (IoU), Dice Coefficient (Dice), and Normalized Surface Distance (NSD). With the assistance of 46 clinical practitioners, an observer study is conducted to benchmark the U-SAM with human performance and evaluate its clinical applicability. The original new source 398 CT scans and our code are openly available for research. Results Our method achieves Dice of 71.23% for normal rectum and 76.38% for rectal tumor, with IoU of 55.32% and 61.78%, and NSD values of 83.63% and 58.59%, respectively, surpassing state-of-the-art methods. The observer study validates that U-SAM can produce diagnostic results comparable to those of highly experienced doctors in just 3 seconds of inference time (with about 5 minutes for prompt acquisition) in clinical settings. Conclusions The proposed U-SAM offers an efficient and reliable method for segmenting rectal cancer and normal tissue, significantly reducing time in clinical settings and effectively assisting radiologists. We believe this initial exploration in CT-based rectal cancer segmentation will be instrumental for future diagnosis.