Embedding-based unsupervised neuron segmentation with size-aware evaluation in digital pathology

Introduction: Accurate cell segmentation is critical for spatial transcriptomics analysis. In human trigeminal ganglion (hTG), the default Xenium pipeline, based on DAPI staining, frequently over-segments neurons into multiple smaller regions, limiting downstream analysis. Here, we leverage readily available hematoxylin and eosin (H&E) images to develop an approach for neuron boundary detection, together with a size-aware evaluation framework.

Methods: We generated 10X Xenium data with matched H&E images from hTG tissues. Image embeddings were extracted using a foundation model (UNI) without fine-tuning. K-means clustering identified neuron-associated regions, followed by connected component superpixel aggregation and Gaussian mixture model refinement to separate large cells. Performance was evaluated against expert manual annotations using intersection over union (IoU), normalized centroid distance (NCD), and relative area error (RAE), along with standard detection metrics (precision, recall, and F1 score).

Results: were further stratified by cell size quartiles (Q1-Q4).ResultsIoU correlated strongly with NCD, with this relationship becoming more consistent in larger cells, indicating improved localization stability. Small cells (Q1) exhibited greater variability and weaker alignment, while large neurons (Q4) achieved higher IoU and more consistent centroid agreement. Standard metrics showed similar size-dependent trends. However, RAE showed weak association with IoU across all quartiles, indicating persistent boundary and size estimation errors.

Discussion: Stratified analysis revealed size-dependent behavior, with improved localization in larger cells but inconsistent boundary accuracy across all groups. Our approach provides reliable neuron localization without model fine-tuning. Given the inherent variability in manual ground truth, it is effective for practical neuron detection, where approximate boundaries are sufficient for downstream transcript assignment.

Learning Objectives:

1. Apply unsupervised embedding-based methods for neuron segmentation in digital pathology
2. Evaluate segmentation using IoU, NCD, and RAE to capture overlap, localization, and size accuracy
3. Interpret size-dependent performance and assess reliability for downstream spatial analysis