Identifying ChatGPT-written Patient Education Materials Using Text Analysis and Readability

OBJECTIVE

 Artificial intelligence (AI)-based text generators such as Chat Generative Pre-Trained Transformer (ChatGPT) have come into the forefront of modern medicine. Given the similarity between AI-generated and human-composed text, tools need to be developed to quickly differentiate the two. Previous work has shown that simple grammatical analysis can reliably differentiate AI-generated text from human-written text.

STUDY DESIGN

 In this study, ChatGPT was used to generate 25 articles related to obstetric topics similar to those made by the American College of Obstetrics and Gynecology (ACOG). All articles were geared towards patient education. These AI-generated articles were then analyzed for their readability and grammar using validated scoring systems and compared to real articles from ACOG.

RESULTS

 Characteristics of the 25 AI-generated articles included fewer overall characters than original articles (mean 3,066 vs. 7,426; p < 0.0001), a greater average word length (mean 5.3 vs. 4.8; p < 0.0001), and a lower Flesch-Kincaid score (mean 46 vs. 59; p < 0.0001). With this knowledge, a new scoring system was develop to score articles based on their Flesch-Kincaid readability score, number of total characters, and average word length. This novel scoring system was tested on 17 new AI-generated articles related to obstetrics and 7 articles from ACOG, and was able to differentiate between AI-generated articles and human-written articles with a sensitivity of 94.1% and specificity of 100% (Area Under the Curve [AUC] 0.99).

CONCLUSION

 As ChatGPT is more widely integrated into medicine, it will be important for health care stakeholders to have tools to separate originally written documents from those generated by AI. While more robust analyses may be required to determine the authenticity of articles written by complex AI technology in the future, simple grammatical analysis can accurately characterize current AI-generated texts with a high degree of sensitivity and specificity.

KEY POINTS

· More tools are needed to identify AI-generated text in obstetrics, for both doctors and patients.. · Grammatical analysis is quick and easily done.. · Grammatical analysis is a feasible and accurate way to identify AI-generated text..

Efficacy of a password-protected, pill-dispensing device with mail return capacity to enhance disposal of unused opioids after cancer surgery

12019

Background: Opioid misuse is a public health crisis. Initial opioid exposures often occur post-operatively, and 10% of opioid-naïve patients who undergo cancer surgery subsequently become long-term opioid users. It has been shown that 70% of opioids prescribed post-operatively go unused, but only 9% of unused pills are disposed of appropriately, which increases the risk of unintended use. We evaluated the impact of an inexpensive, password-protected pill-dispensing device with mail return capacity on disposal of unused pills after cancer surgery. Methods: We conducted a prospective, proof-of-concept pilot study among adult patients scheduled for major cancer-related surgery. Enrolled patients received opioid prescriptions in a pill-dispensing device (Addinex) from a specialty pharmacy. The mechanical device linked to a smartphone app, which provided passwords on a prescriber-defined schedule. Patients were able to enter unique passwords into the device to receive their pills if the prescribed time had elapsed. The smartphone app provided clinical guidance based on patient-reported pain levels, and suggested tapering strategies. Patients were instructed to return the device in a DEA-approved mailer when opioid use was no longer required for acute pain control. Unused pills were destroyed upon receipt. The primary objective was to determine the feasibility of device return, defined as > 50% within 6 weeks. We also explored total pill use and return, patterns of device use and patient satisfaction. Results: We enrolled 30 patients between October 2020 and December 2021. The median age was 46 (range 29–72). Surgical procedures included abdominal hysterectomy (13), mastectomy and reconstruction (10), and soft tissue tumor resections (7). Overall, the majority of participants (n = 24, 80%) returned the device, and more than half (n = 17, 57%) returned the device within 6 weeks of surgery. There were 19 patients who obtained opioids from the device. Among these patients, the majority were satisfied with the device (n = 14, 74%); felt the benefits of the device justified the added steps involved (n = 14, 74%); and would sign up to receive opioids in the device again (n = 13, 68%). The other 11 patients used no opioids. None of these non-users reported any opioid requirements for pain control, and all but one (n = 10, 91%) returned the device and unused pills. In total, 567 opioids were prescribed, and 170 (30%) were used. Of the 397 excess pills, 332 (84% of unused pills, 59% of all pills prescribed) were returned by mail. Conclusions: We found that use of an inexpensive pill-dispensing device with mail return capacity was a feasible and effective strategy to enhance disposal of unused post-operative opioids. Interestingly, a substantial number of prescribed pills were unused. This system also improves confidence with indicated opioid use while reducing diversion.

Abstract P4-11-16: Feasibility and patient satisfaction with a smartphone application to improve medication adherence among patients with breast cancer

Introduction: Low medication adherence is associated with worse outcomes among patients with breast cancer (BC). Retrospective studies have shown decreased survival in patients with BC who are non-adherent to endocrine therapy, but less is known about newer oral cancer therapies and the impact on adherence to medications for chronic non-cancer conditions. New strategies to improve global medication adherence are needed. We assessed the feasibility and patient satisfaction of the Medisafe smartphone application in the initial phase of a trial to increase global medication adherence. Methods: Patients treated with oral anti-neoplastic therapy for treatment of BC were eligible. Enrolled patients received individualized instruction on Medisafe app installation and use. The Medisafe app sends patients push notification reminders at the times specified for each of their standing oral medications. Patients self-report whether each medication dose was taken or missed directly in the app. Patients were followed for 12 weeks. The primary endpoint was feasibility, defined as completion of the 12-week study intervention. uMARS (end-user Mobile Application Rating Scale) questionnaires were used to assess patient satisfaction with the Medisafe app at week 12. The objective feedback component of the uMARS contains 16 questions and includes 4 objective quality subscales: engagement, functionality, aesthetics, and information quality. Each question is answered on a 1-5 scale (5 indicates highest satisfaction), and mean responses were used to calculate an overall satisfaction score and a satisfaction score for each subsection. Results: Between July 2020 and July 2021, 58 patients were enrolled. The mean participant age was 58 years (range 31 - 87). As of June 30,2020 20 patients reached the 12-week endpoint. Of these patients, 18 (90%) completed the 12-week intervention. Of the 17 patients who completed the uMARS questionnaire, the mean overall uMARS score of the Medisafe app was 3.8 (SD 0.6). Table 1 displays the results for the uMARS subscales, which include a mean functionality score of 4.0 (SD 0.7) and mean information quality score of 4.1 (SD 0.7). Notably, 70.6% of participants reported that the app was easy to use and clearly designed, and 76.5% reported that the app’s content was appropriately designed for its specific target audience. Of the 13 patients who reported viewing educational information within the app, 84.6% reported that the information seemed credible. Almost half of patients (43.8%) reported that the app increased their awareness of the importance of addressing health behavior. Conclusions: Our early findings suggest that the use of a smartphone application to improve medication adherence among patients with cancer is feasible, with a high rate of participant completion. Furthermore, high patient-reported satisfaction with the Medisafe app suggests acceptability to promote long-term behavior change. Table 1: Patient Reported Satisfaction with the Medisafe App Evaluated Assessed by the Objective Feedback Component of the uMARS (end-user Mobile Application Rating Scale) Questionnaire at Week 12 (n=17)

uMARS Objective ScoreMean± SDOverall3.8± 0.6  Engagement3.2± 0.8  Functionality4.0± 0.7  Performance4.2± 0.8  Ease of use4.1± 1.0  Aesthetics4.0± 0.6  Information Quality4.1± 0.7  Quality of info3.8± 0.8  Visual info4.3± 0.9  Credibility4.5± 0.8

Citation Format: Melissa K Accordino, Sophie Ulene, Erin Honan, Meghna S Trivedi, Katherine D Crew, Erik Harden, Cynthia Law, Dawn L Hershman. Feasibility and patient satisfaction with a smartphone application to improve medication adherence among patients with breast cancer [abstract]. In: Proceedings of the 2021 San Antonio Breast Cancer Symposium; 2021 Dec 7-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2022;82(4 Suppl):Abstract nr P4-11-16.

Abstract P4-11-33: Continuous glucose monitoring and hyperglycemia during chemotherapy for early-stage breast cancer

Background: There are a growing number of breast cancer (BC) survivors who are at risk for short and long-term treatment-related toxicities. BC survivors may be at higher risk of developing diabetes mellitus (DM), and chemotherapy may potentiate this risk due to concurrent corticosteroid use. DM is associated with both short and long-term treatment toxicities and worse BC outcomes. The prevalence of hyperglycemia (HG) during chemotherapy for early-stage BC (ESBC), and the association between HG during chemotherapy and treatment-related toxicities is unknown. Methods: We are conducting a single-arm pilot study to evaluate the prevalence of hyperglycemia among patients with ESBC during chemotherapy (NCT04473378). Patients are eligible if ≥18 years old, initiating chemotherapy with corticosteroid use, not receiving systemic steroids except as supportive care for chemotherapy, and known DM is allowed if patients are not treated with insulin. Within 7 days of chemotherapy initiation, the Freestyle Libre Pro (Abbott Diabetes Care) continuous glucose monitoring system is applied to the posterior arm of each participant. Patches are reapplied every 2-3 weeks and worn continuously until chemotherapy completion (duration per regimen). The Freestyle Libre Pro monitors interstitial glucose every 15 minutes via subcutaneous sensor filaments adhered to the skin without a finger prick. Data is downloaded using a wireless scan of the sensor by a reader. The primary endpoints are: 1) the prevalence of HG, defined as the number of participants who have ≥1 glucose value (fasting or non-fasting) of ≥140 mg/dL at any point from chemotherapy initiation to completion; 2) Among participants who develop HG, the proportion of time in which they have HG, measured as the number of hyperglycemic values (glucose value of ≥140 mg/dL) divided by the total number of glucose values recorded in an individual for the duration of chemotherapy. Secondary endpoints include the prevalence of impaired glucose tolerance (hemoglobin A1c [HgbA1c] ≥5.7%) prior to chemotherapy initiation in patients without a history of DM, and changes in glucose biomarkers (HgbA1c, fructosamine, and serum creatinine) during treatment. Results: Between December 2020 and April 2021, 7 patients were enrolled, with evaluable data for 5 patients. At baseline median age was 60 (range, 37-74) and median BMI was 33.0 (range, 24.6-41.6). Chemotherapy regimens were: docetaxel/cyclophosphamide (40%); docetaxel/cyclophosphamide/trastuzumab/pertuzumab (20%); weekly paclitaxel (20%); and paclitaxel followed by doxorubicin/cyclophosphamide (20%). All patients (100%) developed hyperglycemia. Of 18,768 sensor readings (281,265 minutes) the proportion of time participants were hyperglycemic (≥140 mg/dL) during the period of adjuvant/neoadjuvant chemotherapy was 22.1%, and the mean time from first corticosteroid administration to first hyperglycemic episode was 7.6 hours. Of three patients with no history of DM (including one patient with glucose intolerance), the proportion of time spent hyperglycemic (≥140 mg/dL) was 9.9% (range, 1.7-13.8%), and the mean daily glucose was 106.7 mg/dL (SD 10.3). Of the two patients with DM, the proportion of time spent hyperglycemic was 70.2% (range, 67.2-79.5%) and the mean daily glucose was 171.7 mg/dL (SD 3.1). Changes in glucose biomarkers will be presented with the full cohort. Conclusion: Hperglycemia during chemotherapy occurred in 100% of the cohort, including those without a history of DM or glucose intolerance. It is currently unknown if HG during chemotherapy is a modifiable risk factor for short and long-term BC treatment toxicities including neuropathy. Understanding glucose trends in this setting will help determine a successful intervention for HG during chemotherapy which may reduce short and long-term treatment toxicities.

Citation Format: Melissa K Accordino, John H Spivack, Sophie Ulene, Erin Honan, Meghna S Trivedi, Katherine D Crew, Erik Harden, Cynthia Law, Dawn L Hershman. Continuous glucose monitoring and hyperglycemia during chemotherapy for early-stage breast cancer [abstract]. In: Proceedings of the 2021 San Antonio Breast Cancer Symposium; 2021 Dec 7-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2022;82(4 Suppl):Abstract nr P4-11-33.

Efficacy of a password-protected pill-dispensing device to enhance disposal of unused opioids after cancer surgery

264

Background: Opioid misuse is a public health crisis. Initial opioid exposures often occur post-operatively, and 10% of opioid-naïve patients who undergo cancer surgery subsequently become long-term opioid users. It has been shown that 70% of opioids prescribed post-operatively go unused, but only 9% of unused pills are disposed appropriately, which increases the risk of unintended use. We evaluated the impact of an inexpensive, password-protected dispensing device with mail return capacity on retrieval of unused pills after cancer surgery. Methods: Adult patients scheduled for major cancer-related surgery were eligible. Enrolled patients received opioid prescriptions in a password-protected, pill-dispensing device (Addinex) from a specialty pharmacy. The mechanical device links to a smartphone app, which provides passwords on a prescriber-defined schedule. Patients request a password when they are in pain, enter the password into the device and receive a pill if the prescribed time has elapsed. The smartphone app provides clinical guidance based on patient-reported pain levels, and suggests tapering strategies. Patients are instructed to return the device in a DEA-approved mailer when opioid use is no longer required for pain control. Unused pills are destroyed upon receipt. The primary objective was to determine the feasibility of device return, defined as > 50% of patients with device return. We also explored patterns of device use, patient reported outcomes, and device satisfaction via surveys and semi-structured interviews. Results: Between October, 2020 and April, 2021, 13 patients completed the study; 4 patients are currently enrolled. Among the initial 13 patients, 7 underwent abdominal hysterectomy, 4 underwent mastectomy and 2 underwent cutaneous tumor resections. The majority of these patients (n = 10, 77%) returned the device, and more than half (n = 7, 54%) returned the device within 6 weeks of surgery. Only a minority of patients (n = 5, 38%) used the device to obtain opioids; most (n = 8, 62%) used no opioids at home, and all of these patients returned the device and the unused pills. Of 11 patients who participated in semi-structured interviews, most (n = 7, 64%) said they felt safer having opioids in the device instead of a regular pill bottle. Among device users, the majority (n = 4, 80%) reported an overall positive experience. All non-users reported having no opioid requirement for pain control. Conclusions: Our early findings suggest that use of an inexpensive, password-protected, pill-dispensing device to assist with opioid dispensing and return is feasible, with a high rate of device and unused opioid return to the pharmacy. This strategy may be effective for reducing opioid diversion. Analyses and recruitment are ongoing to evaluate the benefits of reducing post-operative opioid consumption.