Clinicians usually refer to a diverse array of published articles,  textbooks, personal notes, and online resources for diagnostic decision-making, particularly in complex cases. However, the vast volume of available data often exceeds human memory capacity, leading to inadvertent loss of critical information over time. While recent advancements in Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) based systems enable a first pass search but fail to capture the depth and interconnected nature of medical knowledge required for accurate diagnostics. Dermatopathology is the perfect example as dermatopathologists often refer to not only pathology references but also to a lot of dermatology resources. In our study we use the integration of LLMs and Knowledge Graphs (KGs) for dermatopathology resources to highlight the enhancement of diagnostic capabilities through the structured understanding and dynamic processing of the vast resources of data as compared to a RAG based system.
. 
This presentation highlights an advanced AI framework that combines the precision of KGs with the contextual comprehension of LLMs to create superior query tools for dermatopathology knowledge. We opted for KGs over traditional RAG systems due to their advanced ability to structure and interconnect vast arrays of data. This structuring not only links directly queried information but also intelligently surfaces related concepts and context not explicitly mentioned in the search terms, significantly enhancing the depth and relevance of insights for improved diagnostic accuracy. We compare query tools using the RAG model with LLM with those using KGs  with LLMs. Our approach harnesses four key advantages of KG-based systems:

1. Structured Contextual Understanding: By structuring dermatology and pathology data into a detailed KG that encapsulates diseases, symptoms, histopathological features, and treatments, our AI tool utilizes the LLM’s capabilities to access a rich, interconnected knowledge base for accurate diagnostics, substantially enhancing both specificity and sensitivity.

1. Provenance and Trust: Each diagnostic suggestion by our system is accompanied by provenance, linking back to the data points within the KG that support the diagnosis. This transparency allows clinicians to verify the AI’s recommendations, fostering trust and facilitating wider acceptance in clinical practice.

1. Handling Complex Queries: Leveraging the organized structure of the KG, our system excels at interpreting complex, multi-symptom patient cases that traditional AI systems struggle with. This capability is critical for accurately diagnosing multifaceted dermatological conditions.

1. Efficiency and Resource Management: The use of KGs reduces the need for extensive data traversal during each query, enabling faster response times and reducing computational demands. This efficiency is particularly beneficial in high-volume clinical settings where timely decision-making is crucial.

We used the author’s personal knowledge repository and automatically created a knowledge graph using LLMs as an intermediate tool. Further with LLMs as the interaction layer made extracting information from the KGs simple and distributable. We used the same knowledge repository to create a RAG based system as a baseline comparison. 

For testing we worked with a cohort of 10 dermatopathologists of which 5 worked with a standard RAG+LLM system and 5 with the KG + LLM system, testing for complex questions that required multi-hop associations as in real life drawn from an identical question bank. In the preliminary cohort of 100 questions, the KG+LLM system gave 100% accuracy compared to the RAG+LLM system, which completely failed on multi-hop associations. We will expand the testing with a larger cohort of users as well as an expanded question bank. Future work will focus on continuously expanding the KG to include new research and adapting the LLM integration to incorporate these updates seamlessly.

The implications of our research extend beyond improved diagnostic accuracy, suggesting significant enhancements in clinical workflows, patient outcomes, and the economic aspects of healthcare delivery. This system also facilitates the creation of personalized knowledge graphs for individual practitioners, enhancing their diagnostic capabilities and operational efficiency.

Learning Objectives

1. Understand the different methods of data retrieval using AI.
2. Learn the concept of building a Knowledge Graph from saved data.
3. Harness the power of modern technologies for making better and faster diagnosis.