Extracting diagnoses from hematopathology reports using a large language model

Introduction: Pathology reports present a challenge for structured data extraction given report complexity and the spectrum of potential diagnoses. We assessed a medical domain-specific large language model (LLM) for diagnosis extraction from free-text lymph node pathology reports.

Methods: We acquired the free-text fields from 2,827 pathology reports at Northwestern Memorial Hospital. We developed a dictionary of 145 diagnoses and synonyms for lymphoproliferative and reactive conditions. A pathologist selected 284 reports covering a spectrum of diagnoses and annotated diagnoses with dictionary terms. A privately deployed MedGemma 4B model was prompted to retrieve correct diagnoses from the dictionary per sample in the reports.

Results: From 273 annotated reports, the model returned 394 unique outputs and an additional 110 duplicates that we excluded from the analysis. The correct diagnosis was returned for 273 entries (69%), though 103 were inexact matches to the provided dictionary terms. A total of 25 (6%) were incorrect. In addition, 96 results (25%) were partially incorrect with the following issues: (a) Returning 'negative' for benign/reactive cases instead of a more specific diagnosis (71); (b) Failure to select a dictionary term, instead returning a long description (14); (c) Return of a definitive diagnosis when the report only provided a differential diagnosis (11).

Conclusion: LLMs have potential in search and data retrieval from narrative, complex pathology reports. While promising, the results show challenges in mapping diagnoses to specified terms and inappropriate model confidence not capturing diagnostic uncertainty. MedGemma 4B showed bias towards reciting report text, highlighting potential model context and attention limitations. Future work will focus on improving prompting and context management by enhancing prompts, developing deeper insights into model behavior through ablation studies, and incorporating Retrieval-Augmented Generation techniques.

Learning Objective:

1. Describe the challenges associated with processing pathology reports with large language model
2. Categorize the types of errors seen in processing pathology reports with large language models
3. Describe challenges in the validation of large language model pipelines