|
1
|
Sarkar N, Kumagai M, Meyr S, Pothapragada S, Unberath M, Li G, Ahmed SR, Smith EB, Davis MA, Khatri GD, Agrawal A, Delproposto ZS, Chen H, Caballero CG, Dreizin D. An ASER AI/ML expert panel formative user research study for an interpretable interactive splenic AAST grading graphical user interface prototype. Emerg Radiol 2024; 31:167-178. [PMID: 38302827 DOI: 10.1007/s10140-024-02202-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 01/08/2024] [Indexed: 02/03/2024]
Abstract
PURPOSE The AAST Organ Injury Scale is widely adopted for splenic injury severity but suffers from only moderate inter-rater agreement. This work assesses SpleenPro, a prototype interactive explainable artificial intelligence/machine learning (AI/ML) diagnostic aid to support AAST grading, for effects on radiologist dwell time, agreement, clinical utility, and user acceptance. METHODS Two trauma radiology ad hoc expert panelists independently performed timed AAST grading on 76 admission CT studies with blunt splenic injury, first without AI/ML assistance, and after a 2-month washout period and randomization, with AI/ML assistance. To evaluate user acceptance, three versions of the SpleenPro user interface with increasing explainability were presented to four independent expert panelists with four example cases each. A structured interview consisting of Likert scales and free responses was conducted, with specific questions regarding dimensions of diagnostic utility (DU); mental support (MS); effort, workload, and frustration (EWF); trust and reliability (TR); and likelihood of future use (LFU). RESULTS SpleenPro significantly decreased interpretation times for both raters. Weighted Cohen's kappa increased from 0.53 to 0.70 with AI/ML assistance. During user acceptance interviews, increasing explainability was associated with improvement in Likert scores for MS, EWF, TR, and LFU. Expert panelists indicated the need for a combined early notification and grading functionality, PACS integration, and report autopopulation to improve DU. CONCLUSIONS SpleenPro was useful for improving objectivity of AAST grading and increasing mental support. Formative user research identified generalizable concepts including the need for a combined detection and grading pipeline and integration with the clinical workflow.
Collapse
Affiliation(s)
- Nathan Sarkar
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA
| | - Mitsuo Kumagai
- University of Maryland College Park, 4603 Calvert Rd, College Park, MD, 20740, USA
| | - Samantha Meyr
- University of Maryland College Park, 4603 Calvert Rd, College Park, MD, 20740, USA
| | - Sriya Pothapragada
- University of Maryland College Park, 4603 Calvert Rd, College Park, MD, 20740, USA
| | - Mathias Unberath
- Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD, 21218, USA
| | - Guang Li
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA
| | - Sagheer Rauf Ahmed
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA
- R Adams Cowley Shock Trauma Center, 22 S Greene St, Baltimore, MD, 21201, USA
| | - Elana Beth Smith
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA
- R Adams Cowley Shock Trauma Center, 22 S Greene St, Baltimore, MD, 21201, USA
| | | | | | - Anjali Agrawal
- Teleradiology Solutions, 22 Lianfair Road Unit 6, Ardmore, PA, 19003, USA
| | | | - Haomin Chen
- Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD, 21218, USA
| | | | - David Dreizin
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA.
- R Adams Cowley Shock Trauma Center, 22 S Greene St, Baltimore, MD, 21201, USA.
| |
Collapse
|