CytoSAM: A Confidence-Aware Nuclear Segmentation Model for Bladder Cancer Cytology Assessment

Introduction: Accurate nuclei segmentation in bladder cancer cytology is essential for nuclear-to-cytoplasmic (N:C) ratio assessment under The Paris System (TPS). Current methods optimize for cytoplasmic boundaries but show suboptimal performance with nuclear segmentation-critical for diagnostic reliability in urothelial carcinoma classification.Methods: We present CytoSAM, a novel adaptation of the Segment Anything Model optimized for nuclear segmentation in bladder cytology. The architecture implements a dual-loss optimization function: L_total = α·L_BCE + β·L_IoU-aligned, where L_BCE represents binary cross-entropy for mask prediction and L_IoU-aligned guides confidence score correlation with segmentation quality. Our training methodology processes multiple instance segmentation targets (obtained using Micro-SAM) concurrently from single images, with point prompts derived from erosion on these instance masks. This approach requires minimal human-in-the-loop intervention and eliminates the need for expert annotation, as low-quality annotations were automatically discarded. Inference employs adaptive point prompting with intensity-based detection followed by confidence-filtered multimask generation. The model learns structural relationships between adjacent cellular components and their respective boundaries, outputting predicted masks and their corresponding confidence scores for each instance.Results: Evaluated on Hologic ThinPrep-prepared bladder specimens imaged using the Genius™ Digital Diagnostics Imager, CytoSAM achieved the highest performance with IoU=0.81, Dice=0.89, and F1=0.89, significantly outperforming all other models tested. SegFormer (IoU=0.76, Dice=0.85, F1=0.86) and UNet with attention gates (IoU=0.77, Dice=0.86, F1=0.87) showed comparable performance to each other but fell short of CytoSAM's metrics. Pre-trained, generalist state-of-the-art segmentation models demonstrated lower performance in this specific cytology application, with Cellpose-Nuclei (IoU=0.58, Dice=0.72, F1=0.72) and MicroSAM (IoU=0.53, Dice=0.68, F1=0.68) achieving substantially reduced accuracy metrics despite their proven effectiveness in other contexts. We implemented an advanced version of UNet with attention gates, residual connections, deep supervision, and spatial dropout alongside a SegFormer using efficient attention, overlapping patch embedding, and multi-scale feature fusion. We tested against human-annotated images using Napari and SAM to obtain expert segmentations.Conclusion: CytoSAM enables precise N:C ratio calculation essential for TPS-based risk stratification. The model's confidence-aware approach provides reliability metrics critical for clinical implementation, enhancing computer-assisted diagnosis through standardized morphological assessment.

Learning Objectives: 

1. Identify the potential benefits of CytoSAM as a new approach to nuclear segmentation in bladder cytology compared to current methods.

2. Understand how CytoSAM enables precise N:C ratio calculation through dual-loss optimization, enhancing Paris System risk stratification in bladder cancer diagnosis.

3. Explain the integration of CytoSAM with Hologic's Geniusâ„¢ Digital Diagnostics platform and its potential role in morphological assessment standardization.