Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms

BACKGROUND

The widespread diffusion of Artificial Intelligence (AI) platforms is revolutionizing how health-related information is disseminated, thereby highlighting the need for tools to evaluate the quality of such information. This study aimed to propose and validate the Quality Assessment of Medical Artificial Intelligence (QAMAI), a tool specifically designed to assess the quality of health information provided by AI platforms.

METHODS

The QAMAI tool has been developed by a panel of experts following guidelines for the development of new questionnaires. A total of 30 responses from ChatGPT4, addressing patient queries, theoretical questions, and clinical head and neck surgery scenarios were assessed by 27 reviewers from 25 academic centers worldwide. Construct validity, internal consistency, inter-rater and test-retest reliability were assessed to validate the tool.

RESULTS

The validation was conducted on the basis of 792 assessments for the 30 responses given by ChatGPT4. The results of the exploratory factor analysis revealed a unidimensional structure of the QAMAI with a single factor comprising all the items that explained 51.1% of the variance with factor loadings ranging from 0.449 to 0.856. Overall internal consistency was high (Cronbach's alpha = 0.837). The Interclass Correlation Coefficient was 0.983 (95% CI 0.973-0.991; F (29,542) = 68.3; p < 0.001), indicating excellent reliability. Test-retest reliability analysis revealed a moderate-to-strong correlation with a Pearson's coefficient of 0.876 (95% CI 0.859-0.891; p < 0.001).

CONCLUSIONS

The QAMAI tool demonstrated significant reliability and validity in assessing the quality of health information provided by AI platforms. Such a tool might become particularly important/useful for physicians as patients increasingly seek medical information on AI platforms.

Impact of Tracheostomy Timing Within the National Veterans Affairs Population

OBJECTIVE

There is a lack of a definitive study in the literature comparing early versus late tracheostomy and exploring the impact of tracheostomy timing on patient outcomes. This study may help guide treatment paradigms and contribute to a consensus for optimal tracheostomy timing.

METHODS

A retrospective review was performed comparing early versus late timing of tracheostomy placement and their respective outcomes. The authors used data provided by VA Informatics and Computing Infrastructure (VINCI) to find patients who received a tracheostomy at any VA Medical Center in the United States. There were a total of 25,334 tracheostomies in the database which satisfied our criteria. These occurred between the years 1999 and 2022. Propensity score matching assessed 17,074 tracheostomies, 8537 in either group. The median age of patients in the matched groups was 66 years, and approximately 97.4% of patients were male. Early tracheostomy timing was defined as the placement of the tracheostomy within 10 days of intubation. Outcomes included post-tracheostomy intensive care unit (ICU) days, post-tracheostomy hospital days, successful ventilator weaning, and all-cause mortality.

RESULTS

Early tracheostomy was associated with significantly fewer ICU days and hospital days, and the early group experienced higher rates of successful ventilator weaning. Survival analysis of data within 5 years of tracheostomy showed that early tracheostomy was associated with significantly lower hazard for all-cause mortality.

CONCLUSION

Our results add to the body of evidence that an earlier transition to mechanical ventilation by tracheostomy confers benefits in patient morbidity and mortality as well as resource utilization.

LEVEL OF EVIDENCE

3 Laryngoscope, 2024.

Comparative impact of COVID-19 infection on tracheostomy patients

PURPOSE

We study outcomes after tracheostomy in COVID-19 positive patients versus COVID-19 negative patients who underwent tracheostomy during the same time frame in an effort to better understand the influence of COVID-19 despite variances in virus strain and treatment practices.

MATERIALS AND METHODS

This is a retrospective cohort study of all Veterans Affairs centers nationwide, using data provided by the Veterans Affairs Informatics and Computing Infrastructure. Our cohort consisted of veteran patients who underwent tracheostomy between March 2020 and September 2022. Patients who tested positive for COVID-19 within three months prior to tracheostomy were compared to patients who had never tested positive for COVID-19.

RESULTS

956 patients were included in the analysis, and nearly 96 % of these patients were male. The COVID-19 positive group spent one more week on the ventilator and experienced lower rates of successful ventilator weaning (hazard ratio 0.74, 95 % confidence interval [0.62, 0.88], P < 0.001). Survival curves were non-proportional, and while the COVID-19 positive group had higher 30-day mortality (relative risk 1.37, 95 % confidence interval [1.09, 1.73], P = 0.007), the COVID-19 negative group had higher long-term mortality.

CONCLUSIONS

Our findings suggest that while infection with COVID-19 has a significant effect on short-term outcomes after tracheostomy, chronic comorbidities seem to have the more enduring impact. In spite of prolonged ventilation and higher short-term mortality, tracheostomy in COVID-19 can be a positive intervention that does not necessarily predestine patients to the same level of long-term morbidity and mortality of typical tracheostomies.