Pathology image classification is improved through combining vision and text model embedding inputs

Background: Morphological descriptions of pathology specimens capture nuances beyond standalone diagnoses. Pathology models incorporating language may enhance and complement the predictive ability of vision-only models and better represent how pathologists utilize morphologic details. We applied this approach to the classification of dermatopathology (Derm) and gastrointestinal (GI) specimens, two areas with challenging diagnosis due to a high number of related entities and rare conditions.

Methods: In a cohort of H&E-stained Derm (N=29,655) and GI (N=59,035) whole slide images, slide-level embeddings were learned from the PLUTO vision foundation model (PathAI, Boston, MA). Associated text descriptions were obtained from diagnostic reports and embedded using a general purpose language foundation model and learned projection layer. Slide and text embeddings were aligned using a contrastive learning objective. Performance of multimodal models was compared to vision-only additive multiple instance learning (aMIL) approaches.

Results: Compared to aMIL models, multimodal models demonstrated a 4-6% relative improvement in class accuracy, subclass accuracy, and subclass-weighted F1 for Derm and an 8-10% relative improvement for GI (Fig. 1). Visualization of embeddings revealed that text and vision embeddings clustered closely in slides with related class labels, reflecting alignment between image features and clinical descriptions.

Conclusions: Diagnostic pathologists interpret patterns, context, and relationships across a slide. By incorporating language into AI-powered pathology, models shift from solely reasoning over visual patterns to reasoning over descriptive disease representations, potentially enabling more adaptable systems to meet the complexity and variability of real-world pathology. By combining strong visual foundation model outputs with ongoing advances in language models, multimodal approaches represent a promising direction for the future of digital pathology.

Learning Objectives:

1. Understand how combining language model and vision model embeddings can improve AI-powered pathology image classification.