A scalable, standardized, and responsible framework for the translation and deployment of clinical AI models
Background: The development, implementation, and maintenance of artificial intelligence and machine learning (AI/ML) systems are complex endeavors - applying them to the clinical care of patients greatly increases that complexity. The non-negotiable nature of medicine demands clinical AI/ML systems follow specific key characteristics, including that they be safe, socially responsible, transparent, closely monitored, and follow regulatory guidelines. Unfortunately, the circuitous nature of AI model training, translation, operationalization, and maintenance does not follow typical healthcare information technology (IT) or clinical laboratory deployments. Therefore, we found the need to develop a specialized AI lifecycle framework to operationally account for this new complexity, manage expectations for AI deployment amongst our pathologists and laboratory staff, and ensure the aforementioned key characteristics were integrated into AI/ML solutions deployed within Laboratory Medicine and Pathology.
Methods: A literature search was performed and internal processes for AI/ML system development, test development and information system deployment were reviewed. An initial AI lifecycle framework was developed, upon which multiple internal discussions and iterations occurred. An enterprise stakeholder analysis was then performed, with key stakeholders external to the department of laboratory medicine and pathology consulted to further develop and refine the lifecycle framework. A final AI lifecycle framework was then developed for both internal and external socialization, feedback, and ultimately, deployment.
Results: Our AI lifecycle framework is based on the concept of Technology Readiness Levels and adapted for clinical AI/ML system deployment from the published (2022) Nature Communications article “Technology Readiness Levels for Machine Learning Systems”. The framework is designed to capture AI/ML solutions at any stage of development, from initial concepts (what problem are you trying to solve) to formal clinical AI/ML model go-live and system maintenance. Ultimately, the goal of the AI lifecycle framework is to standardize the different paths and documentation needed for clinical AI/ML development and deployment (regardless of platform) by tracking and promoting system maturity at each technology readiness level in the following workstreams: clinical use case, data, data pipeline, code status (AI/ML), clinical user experience, clinical technology infrastructure, clinical orchestration, regulatory compliance, and project management. Using this framework, one can begin to formalize the process of developing, evaluating, and deploying a range of AI/ML models and systems (from internally developed to FDA-cleared models).
Conclusions: We have created and established a novel lifecycle framework for clinical AI/ML systems in Pathology and Laboratory Medicine that can reduce the inherent complexity, improve scalability and transparency, manage user expectations, and promote high-quality characteristics in the clinical development, implementation, and maintenance of these systems.
- Recognize the complexity of clinical AI/ML development, deployment and maintenance
- Discuss the utility of Technology Readiness Levels in the development of AI/ML systems
- Understand how to adapt the AI Lifecycle Framework for their institution’s needs to promote AI model adoption
Dave McClintock, MD
Chair, Division of Computational Pathology and AI
David McClintock, MD, is the Chair of the Division of Computational Pathology and Artificial Intelligence within the Department of Laboratory Medicine and Pathology at Mayo Clinic (Rochester, MN). His primary clinical interests include clinical informatics, laboratory workflow optimization, digital pathology implementation, analytics, and clinical machine learning/AI model deployment. His research interests include understanding the role and effects of digital pathology within the clinical laboratories and the use of artificial intelligence and machine learning for improved diagnostics, more efficient workflows, and better patient outcomes.
Chris Garcia, MD, MS
Medical Director, Division of Computational Pathology and AI
Dr. Chris Garcia is the Medical Director in the Division of Computational Pathology and Artificial Intelligence at the Mayo Clinic. He is a pathologist (AP/CP) with subspecialty training in Pathology Informatics. He has over10 years experience in working in industry (Philips Digital Pathology Solutions), for reference laboratories (Labcorp as a medical director and strategic director) and academic medicine (currently at the Mayo Clinic). He is actively involved in the digital pathology and pathology informatics communities. He currently focuses on developing and integrating AI/ML solutions into clinical operations and practice in Laboratory Medicine and Pathology.