1
|
Mourits BMP, den Hartog SJ, de Graaf JA, Roozenbeek B, Post MWM, Visser-Meily JMA, Scholten EWM. Exploring patients' experience using PROMs within routine post-discharge follow-up assessment after stroke: a mixed methods approach. J Patient Rep Outcomes 2024; 8:46. [PMID: 38668901 PMCID: PMC11052975 DOI: 10.1186/s41687-024-00724-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Accepted: 04/17/2024] [Indexed: 04/29/2024] Open
Abstract
BACKGROUND Patient Reported Outcomes Measures (PROMs) are being used increasingly to measure health problems in stroke clinical practice. However, the implementation of these PROMs in routine stroke care is still in its infancy. To understand the value of PROMs used in ischemic stroke care, we explored the patients' experience with PROMs and with the consultation at routine post-discharge follow-up after stroke. METHODS In this prospective mixed methods study, patients with ischemic stroke completed an evaluation questionnaire about the use of PROMs and about their consultation in two Dutch hospitals. Additionally, telephone interviews were held to gain in-depth information about their experience with PROMs. RESULTS In total, 63 patients completed the evaluation questionnaire of which 10 patients were also interviewed. Most patients (82.2-96.6%) found completing the PROMs to be feasible and relevant. Half the patients (49.2-51.6%) considered the PROMs useful for the consultation and most patients (87.3-96.8%) reported the consultation as a positive experience. Completing the PROMs provided 51.6% of the patients with insight into their stroke-related problems. Almost 75% of the patients found the PROMs useful in giving the healthcare provider greater insight, and 60% reported discussing the PROM results during the consultation. Interviewed patients reported the added value of PROMs, particularly when arranging further care, in gaining a broader insight into the problems, and in ensuring all important topics were discussed during the consultation. CONCLUSIONS Completing PROMs appears to be feasible for patients with stroke attending post-discharge consultation; the vast majority of patients experienced added value for themselves or the healthcare provider. We recommend that healthcare providers discuss the PROM results with their patients to improve the value of PROMs for the patient. This could also improve the willingness to complete PROMs in the future.
Collapse
Affiliation(s)
- B M P Mourits
- Center of Excellence for Rehabilitation Medicine, UMC Utrecht Brain Center, University Medical Center Utrecht, and De Hoogstraat Rehabilitation, Utrecht, The Netherlands.
| | - S J den Hartog
- Department of Neurology, Erasmus MC University Medical Center, Rotterdam, The Netherlands
| | - J A de Graaf
- Center of Excellence for Rehabilitation Medicine, UMC Utrecht Brain Center, University Medical Center Utrecht, and De Hoogstraat Rehabilitation, Utrecht, The Netherlands
- Department of Rehabilitation, Physical Therapy Science and Sports, UMC Utrecht Brain Center, University Medical Center, Utrecht, The Netherlands
| | - B Roozenbeek
- Department of Neurology, Erasmus MC University Medical Center, Rotterdam, The Netherlands
| | - M W M Post
- Center of Excellence for Rehabilitation Medicine, UMC Utrecht Brain Center, University Medical Center Utrecht, and De Hoogstraat Rehabilitation, Utrecht, The Netherlands
- University of Groningen, University Medical Center Groningen, Department of Rehabilitation Medicine, Groningen, The Netherlands
| | - J M A Visser-Meily
- Center of Excellence for Rehabilitation Medicine, UMC Utrecht Brain Center, University Medical Center Utrecht, and De Hoogstraat Rehabilitation, Utrecht, The Netherlands
- Department of Rehabilitation, Physical Therapy Science and Sports, UMC Utrecht Brain Center, University Medical Center, Utrecht, The Netherlands
| | - E W M Scholten
- Center of Excellence for Rehabilitation Medicine, UMC Utrecht Brain Center, University Medical Center Utrecht, and De Hoogstraat Rehabilitation, Utrecht, The Netherlands
| |
Collapse
|
2
|
Henderson NL, Basch EM, Rocque GB. Learning to Overcome Barriers to Electronic Patient-Reported Outcome Implementation: Lessons From the Field. JCO Oncol Pract 2024:OP2400104. [PMID: 38502891 DOI: 10.1200/op.24.00104] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2024] [Accepted: 02/14/2024] [Indexed: 03/21/2024] Open
Abstract
#Implementation of #ePROs in real world settings, lessons learned.
Collapse
Affiliation(s)
- Nicole L Henderson
- University of Alabama at Birmingham, Department of Medicine, Division of Hematology and Oncology, Birmingham, AL
| | - Ethan M Basch
- Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Gabrielle B Rocque
- University of Alabama at Birmingham, Department of Medicine, Division of Hematology and Oncology, Birmingham, AL
- University of Alabama at Birmingham, Department of Medicine, Division of Gerontology, Geriatrics, and Palliative Care, Birmingham, AL
- O'Neal Comprehensive Cancer Center, Birmingham, AL
| |
Collapse
|
3
|
Mohindra NA, Coughlin A, Kircher S, O'Daniel A, Barnard C, Cameron KA, Hirschhorn LR, Cella D. Implementing a Patient-Reported Outcome Dashboard in Oncology Telemedicine Encounters: Clinician and Patient Adoption and Acceptability. JCO Oncol Pract 2024; 20:409-418. [PMID: 38207229 DOI: 10.1200/op.23.00493] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Revised: 10/16/2023] [Accepted: 11/16/2023] [Indexed: 01/13/2024] Open
Abstract
PURPOSE Telemedicine provides numerous benefits to patients, yet effective communication and symptom assessment remain a concern. The recent uptake of telemedicine provided an opportunity to use a newly developed dashboard with patient-reported outcome (PRO) information to enhance communication and shared decision making (SDM) during telemedicine appointments. The objective of this study was to identify barriers to using the dashboard during telemedicine, develop implementation strategies to address barriers, and pilot test use of this dashboard during telemedicine appointments in two practice settings to evaluate acceptability, adoption, fidelity, and effectiveness. METHODS Patients and clinicians were interviewed to identify determinants to dashboard use in telemedicine. Implementation strategies were designed and refined through iterative feedback from stakeholders. A pilot study of dashboard use was conducted from March to September 2022. Acceptability, adoption, and fidelity were evaluated using mixed methods. SDM was evaluated using the collaboRATE measure. RESULTS One hundred two patient encounters were evaluated. Most patients (62; 60%) had completed some PRO data at the time of their telemedicine encounter. Most (82; 80%) encounters had clinician confirmation that PRO data had been reviewed; however, collaborative review of the dashboard was documented in only 27%. Degree of SDM was high (mean collaboRATE score 3.40; SD, 0.11 [95% CI, 3.17 to 3.63] out of a maximum score of 4). Implementation strategies focused on patient engagement, education, and remote PRO completion. Clinician-facing strategies included education, practice facilitation, and small tests of change. CONCLUSION This study demonstrated that implementation of a PRO-based dashboard into telemedicine appointments was feasible and had acceptable adoption and acceptability by patients and clinicians when several strategies were used to engage end users. Strategies targeting both patients and clinicians are needed to support routine and effective PRO integration in telemedicine.
Collapse
Affiliation(s)
- Nisha A Mohindra
- Division of Hematology and Oncology, Department of Medicine, Feinberg School of Medicine, Northwestern University, Chicago, IL
- Robert H. Lurie Comprehensive Cancer Center, Northwestern University, Chicago, IL
| | - Ava Coughlin
- Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, Chicago, IL
| | - Sheetal Kircher
- Division of Hematology and Oncology, Department of Medicine, Feinberg School of Medicine, Northwestern University, Chicago, IL
- Robert H. Lurie Comprehensive Cancer Center, Northwestern University, Chicago, IL
| | | | - Cynthia Barnard
- Northwestern Memorial Health Care, Chicago, IL
- Division of General Internal Medicine, Department of Medicine, Feinberg School of Medicine, Northwestern University, Chicago, IL
| | - Kenzie A Cameron
- Robert H. Lurie Comprehensive Cancer Center, Northwestern University, Chicago, IL
- Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, Chicago, IL
- Division of General Internal Medicine, Department of Medicine, Feinberg School of Medicine, Northwestern University, Chicago, IL
| | - Lisa R Hirschhorn
- Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, Chicago, IL
| | - David Cella
- Robert H. Lurie Comprehensive Cancer Center, Northwestern University, Chicago, IL
- Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, Chicago, IL
| |
Collapse
|
4
|
Lai-Kwon J, Rutherford C, Jefford M, Gore C, Best S. Using Implementation Science Frameworks to Guide the Use of Electronic Patient-Reported Outcome Symptom Monitoring in Routine Cancer Care. JCO Oncol Pract 2024; 20:335-349. [PMID: 38206290 DOI: 10.1200/op.23.00462] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Revised: 11/02/2023] [Accepted: 11/20/2023] [Indexed: 01/12/2024] Open
Abstract
PURPOSE Electronic patient-reported outcomes (ePROs) are an evidence-based means of detecting symptoms earlier and improving patient outcomes. However, there are few examples of successful implementation in routine cancer care. We conducted a qualitative study to identify barriers and facilitators to implementing ePRO symptom monitoring in routine cancer care using the Consolidated Framework for Implementation Research (CFIR). METHODS Participants were adult patients with cancer, their caregivers, or health care professionals involved in ePRO monitoring or processes. Focus groups or individual interviews were conducted using a semistructured approach informed by the CFIR. Data were analyzed deductively using the CFIR. Barriers were matched to theory-informed implementation strategies using the CFIR-Expert Recommendations for Implementing Change (ERIC) matching tool. RESULTS Thirty participants were interviewed: 22 females (73%), aged 31-70 years (28, 94%), comprising patients (n = 8), caregivers (n = 2), medical oncologists (n = 4), nurses (n = 4), hospital leaders (n = 6), clinic administrators (n = 2), pharmacists (n = 2), and information technology specialists (n = 2). Barriers pertaining to four CFIR domains were identified and several were novel, including the challenge of adapting ePROs for different anticancer treatments. Facilitators pertaining to all CFIR domains were identified, such as leveraging acceptability of remote care post-COVID-19 to drive implementation. Conducting consensus discussions with stakeholders to tailor ePROs to the local setting, identifying/preparing individual and group-level champions, and assessing readiness for change (including leveraging technological advances and increased confidence in using remote monitoring post-COVID-19) were the most frequently recommended implementation strategies. CONCLUSION The CFIR facilitated identification of known and novel barriers and facilitators to implementing ePRO symptom monitoring in routine cancer care. Implementation strategies summarized in a conceptual framework will be used to codesign an ePRO symptom monitoring system for immunotherapy side effects.
Collapse
Affiliation(s)
- Julia Lai-Kwon
- Department of Medical Oncology, Peter MacCallum Cancer Centre, Melbourne, Australia
- Department of Health Services Research, Peter MacCallum Cancer Centre, Melbourne, Australia
| | - Claudia Rutherford
- Cancer Care Research Unit (CCRU), Susan Wakil School of Nursing and Midwifery, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia
| | - Michael Jefford
- Department of Medical Oncology, Peter MacCallum Cancer Centre, Melbourne, Australia
- Department of Health Services Research, Peter MacCallum Cancer Centre, Melbourne, Australia
- Australian Cancer Survivorship Centre, Peter MacCallum Cancer Centre, Melbourne, Australia
- Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, Australia
| | - Claire Gore
- Department of Psychology, Peter MacCallum Cancer Centre, Melbourne, Australia
| | - Stephanie Best
- Department of Health Services Research, Peter MacCallum Cancer Centre, Melbourne, Australia
- Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, Australia
- Department of Psychology, Peter MacCallum Cancer Centre, Melbourne, Australia
- Australian Genomics, Murdoch Childrens Research Institute, Melbourne, Australia
- Victorian Comprehensive Cancer Centre Alliance, Melbourne, Australia
| |
Collapse
|
5
|
Paudel R, Tramontano AC, Cronin C, Wong SL, Dizon DS, Jenkins HH, Bian J, Osarogiagbon RU, Schrag D, Hassett MJ. Assessing Patient Readiness for an Electronic Patient-Reported Outcome-Based Symptom Management Intervention in a Multisite Study. JCO Oncol Pract 2024; 20:77-84. [PMID: 38011613 PMCID: PMC10827290 DOI: 10.1200/op.23.00339] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 09/08/2023] [Accepted: 10/16/2023] [Indexed: 11/29/2023] Open
Abstract
PURPOSE While the use of electronic patient-reported outcomes (ePROs) in routine clinical practice is increasing, barriers to patient engagement limit adoption. Studies have focused on technology access as a key barrier, yet other characteristics may also confound readiness to use ePROs including patients' confidence in using technology and confidence in asking clinicians questions. METHODS To assess readiness to use ePROs, adult patients from six US-based health systems who started a new oncology treatment or underwent a cancer-directed surgery were invited to complete a survey that assessed access to and confidence in the use of technology, ease of asking clinicians questions about health, and symptom management self-efficacy. Multivariable ordinal logistic regression models were fit to assess the association between technology confidence, ease of asking questions, and symptom management self-efficacy. RESULTS We contacted 3,212 individuals, and 1,043 (33%) responded. The median age was 63 years, 68% were female, and 75% reported having access to patient portals. Over 80% had two or more electronic devices. Most patients reported high technology confidence, higher ease of asking clinicians questions, and high symptom management self-efficacy (n = 692; 66%). Patients with high technology confidence also reported higher ease of asking nurses about their health (adjusted odds ratio [AOR], 4.58 [95% CI, 2.36 to 8.87]; P ≤ .001). Those who reported higher ease of asking nurses questions were more likely to report higher confidence in managing symptoms (AOR, 30.54 [95% CI, 12.91 to 72.30]; P ≤ .001). CONCLUSION Patient readiness to use ePROs likely depends on multiple factors, including technology and communication confidence, and symptom management self-efficacy. Future studies should assess interventions to address these factors.
Collapse
Affiliation(s)
| | | | | | | | - Don S. Dizon
- Lifespan Cancer Institute and Brown University, Providence, RI
| | | | | | | | | | | |
Collapse
|
6
|
Basch E, Hudson K, Rocque G. Implementation of electronic patient-reported outcomes for symptom monitoring during cancer treatment: the importance of getting it right. J Comp Eff Res 2023; 12:e230157. [PMID: 37961992 PMCID: PMC10734319 DOI: 10.57264/cer-2023-0157] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2023] [Accepted: 10/26/2023] [Indexed: 11/15/2023] Open
Affiliation(s)
- Ethan Basch
- Division of Oncology, University of North Carolina, Chapel Hill, NC 27599, USA
| | | | - Gabrielle Rocque
- Division of Hematology & Oncology, University of Alabama at Birmingham, Birmingham, AL 38294, USA
| |
Collapse
|
7
|
Smith KL, Tsai HL, Lim D, Wang C, Nunes R, Wilkinson MJ, Sheng JY, Couzi R, Fetting J, Riley C, Wolff AC, Santa-Maria CA, Papathakis K, Collins-Chase L, Hilton C, Thorner E, Montanari A, Ikejiani D, Snyder C, Stearns V. Feasibility of Symptom Monitoring During the First Year of Endocrine Therapy for Early Breast Cancer Using Patient-Reported Outcomes Collected via Smartphone App. JCO Oncol Pract 2023; 19:981-989. [PMID: 37733984 DOI: 10.1200/op.23.00038] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2023] [Revised: 04/07/2023] [Accepted: 08/18/2023] [Indexed: 09/23/2023] Open
Abstract
PURPOSE Treatment-associated symptoms drive early discontinuation of adjuvant endocrine therapy (ET) for breast cancer. We hypothesized that symptom monitoring with electronic patient-reported outcomes (ePROs) during adjuvant ET will enhance symptom detection, symptom management, and persistence. METHODS Eligible patients were initiating ET for stage 0-III breast cancer. Participants completed ePRO surveys via smartphone at baseline and 1, 3, 6, and 12 months. Measures included Patient-Reported Outcomes Measurement Information System Anxiety, Depression, Fatigue, and Vaginal Discomfort; plus Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events items assessing joint pain, hot flashes, vaginal dryness, concentration problems, and memory problems. Scores surpassing prespecified thresholds triggered alerts, and recommended symptom management pathways were provided to clinicians. The primary objective was to evaluate feasibility, assessed by survey completion rates, with targets of >65% for the baseline survey and ≥1 follow-up survey during the first 6 months. Secondary objectives included 12-month ET discontinuation rate (target: ≤15%), describing symptoms and evaluating pathway implementation. RESULTS Among 250 participants, 73.2% completed the baseline survey and 69.6% completed ≥1 follow-up survey during the first 6 months. Thirty-one percent of participants had ≥1 symptom alert at baseline and 74% had ≥1 symptom alert during follow-up. The proportions of participants for whom pathway-concordant symptom management was documented at each time point ranged from 12.8% to 36.6%. Twenty-eight participants (11.2%) discontinued ET by 12 months. CONCLUSION Symptom monitoring with ePROs during adjuvant ET is feasible. Despite infrequent documentation of pathway-concordant symptom management after symptom alerts, ePROs were associated with favorable short-term ET persistence.
Collapse
Affiliation(s)
- Karen Lisa Smith
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
- AstraZeneca, Gaithersburg, MD
| | - Hua-Ling Tsai
- Division of Biostatistics and Bioinformatics, Johns Hopkins Sidney Kimmel Comprehensive Cancer Center, Baltimore, MD
| | - David Lim
- Division of Biostatistics and Bioinformatics, Johns Hopkins Sidney Kimmel Comprehensive Cancer Center, Baltimore, MD
- Division of Statistics Collaborative Inc, WCG, Washington, DC
| | - Chenguang Wang
- Division of Biostatistics and Bioinformatics, Johns Hopkins Sidney Kimmel Comprehensive Cancer Center, Baltimore, MD
- Regeneron Pharmaceuticals, Tarrytown, NY
| | - Raquel Nunes
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
- AstraZeneca, Gaithersburg, MD
| | - Mary J Wilkinson
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Jennifer Y Sheng
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Rima Couzi
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | - John Fetting
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Carol Riley
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Antonio C Wolff
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Cesar A Santa-Maria
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Katie Papathakis
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | | | - Christie Hilton
- Division of Hematology and Oncology, Allegheny Health Network Cancer Institute, Pittsburgh, PA
| | - Elissa Thorner
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Amanda Montanari
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| | | | - Claire Snyder
- Johns Hopkins Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, MD
- Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD
- Department of Oncology, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Vered Stearns
- Sidney Kimmel Comprehensive Cancer Center, Women's Malignancies Disease Group, Johns Hopkins University School of Medicine, Baltimore, MD
| |
Collapse
|
8
|
Smit Y, Metsemakers SJJPM, Janssen JJWM, Posthuma EFM, Walraven I, Hermens RPMG, Blijlevens NMA. Measuring chronic myeloid leukaemia TKI-related toxic effects in the real world: a systematic review and critical assessment of content validity of patient-reported outcome measures. Lancet Haematol 2023; 10:e849-e859. [PMID: 37604177 DOI: 10.1016/s2352-3026(23)00173-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Revised: 06/02/2023] [Accepted: 06/06/2023] [Indexed: 08/23/2023]
Abstract
Insight into real-world treatment-related toxic effects reported by patients has the potential to improve care, benchmark trials, and fill knowledge gaps, especially in patients with chronic myeloid leukaemia, which is treated in the majority of patients continually with tyrosine-kinase inhibitors (TKIs). The aim of our systematic review was to investigate the content validity of instruments that elicit TKI-related toxic effects reported by patients with chronic myeloid leukaemia in the real world. We searched PubMed and Embase from Jan 1, 2017 to Oct 21, 2022. Studies on instruments used in or developed for patients with chronic myeloid leukaemia that assess a patient's symptoms were eligible. Content validity was assessed according to the Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN): none of the six identified instruments were rated as sufficient. Five instruments (European Organisation for Research and Treatment of Cancer Core Quality of Life Questionnaire for chronic myeloid leukaemia with 24 items [EORTC QLQ-CML24], EORTC symptom set, Functional Assessment of Cancer Therapy-Leukaemia [FACT-LEU], haematological malignancies patient-reported outcomes [HM-PRO], and MD Anderson Symptom Inventory for chronic myeloid leukaemia [MDASI-CML]) were rated as inconsistent due to not being evaluated by professionals post-development, having very few patients with chronic myeloid leukaemia involved, or missing key symptoms. Moderate-quality to very low-quality evidence underpinned these ratings. The two EORTC instruments were the only ones not to miss key toxic effects (eg, muscle cramps). However, their relevance was rated as inconsistent: the QLQ-CML24 includes questions on health-related quality-of-life, whereas the symptom set includes items sourced from solid cancer treatments. This Review shows the need for an instrument with sufficient content validity to measure toxic effects from TKI treatment in patients with chronic myeloid leukaemia. Until then, stakeholders can make an informed choice from currently used instruments with our assessment.
Collapse
Affiliation(s)
- Yolba Smit
- Department of Haematology, Radboud University Medical Center, Nijmegen, Netherlands.
| | | | - Jeroen J W M Janssen
- Department of Haematology, Radboud University Medical Center, Nijmegen, Netherlands
| | - Eduardus F M Posthuma
- Department of Internal Medicine, Reinier de Graaf Hospital, Delft, Netherlands; Department of Haematology, Leiden University Medical Center, Leiden, Netherlands
| | - Iris Walraven
- Department for Health Evidence, Radboud University Medical Center, Nijmegen, Netherlands
| | - Rosella P M G Hermens
- Department of IQ Healthcare, Radboud University Medical Center, Nijmegen, Netherlands
| | | |
Collapse
|
9
|
Sener T, Haenen W, Smits P, Hans GH. Large-scale real-life implementation of technology-enabled care to maximize hospitals' medical surge preparedness during future infectious disease outbreaks and winter seasons: a viewpoint. Front Public Health 2023; 11:1149247. [PMID: 37621607 PMCID: PMC10446840 DOI: 10.3389/fpubh.2023.1149247] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2023] [Accepted: 07/24/2023] [Indexed: 08/26/2023] Open
Abstract
Hospitals can be overburdened with large numbers of patients with severe infectious conditions during infectious disease outbreaks. Such outbreaks or epidemics put tremendous pressure on the admission capacity of care facilities in the concerned region, negatively affecting the elective program within these facilities. Such situations have been observed during the recent waves of the coronavirus disease pandemic. Owing to the imminent threat of a "tripledemic" by new variants of the coronavirus disease (such as the new Omicron XBB.1.16 strain), influenza, and respiratory syncytial virus during future winter seasons, healthcare agencies should take decisive steps to safeguard hospitals' surge capacity while continuing to provide optimal and safe care to a potentially large number of patients in their trusted home environment. Preparedness of health systems for infectious diseases will require dynamic interaction between a continuous assessment of region-wide available hospital capacity and programs for intensive home treatment of patients who can spread the disease. In this viewpoint, we describe an innovative, dynamic coupling system between hospital surge capacity and cascading activation of a nationwide system for remote patient monitoring. This approach was developed using the multi-criteria decision analysis methodology, considering previously published real-life experiences on remote patient monitoring.
Collapse
Affiliation(s)
- Talia Sener
- Faculty of Medicine and Health Sciences, University of Antwerp, Antwerp, Belgium
| | - Winne Haenen
- Federal Public Service for Health, Food Chain Safety and Environment, Brussels, Belgium
| | - Patrick Smits
- Cell Crisis Preparedness, Agentschap Zorg en Gezondheid, Brussels, Belgium
| | - Guy H. Hans
- Faculty of Medicine and Health Sciences, University of Antwerp, Antwerp, Belgium
- Chief Medical Officer, Antwerp University Hospital (UZA), Edegem, Belgium
| |
Collapse
|
10
|
Moreno A, Solanki AA, Xu T, Lin R, Palta J, Daugherty E, Hong D, Hong J, Kamran SC, Katsoulakis E, Brock K, Feng M, Fuller C, Mayo C. Identification of Key Elements in Prostate Cancer for Ontology Building via a Multidisciplinary Consensus Agreement. Cancers (Basel) 2023; 15:3121. [PMID: 37370731 DOI: 10.3390/cancers15123121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2023] [Revised: 05/25/2023] [Accepted: 06/01/2023] [Indexed: 06/29/2023] Open
Abstract
BACKGROUND Clinical data collection related to prostate cancer (PCa) care is often unstructured or heterogeneous among providers, resulting in a high risk for ambiguity in its meaning when sharing or analyzing data. Ontologies, which are shareable formal (i.e., computable) representations of knowledge, can address these challenges by enabling machine-readable semantic interoperability. The purpose of this study was to identify PCa-specific key data elements (KDEs) for standardization in clinic and research. METHODS A modified Delphi method using iterative online surveys was performed to report a consensus agreement on KDEs by a multidisciplinary panel of 39 PCa specialists. Data elements were divided into three themes in PCa and included (1) treatment-related toxicities (TRT), (2) patient-reported outcome measures (PROM), and (3) disease control metrics (DCM). RESULTS The panel reached consensus on a thirty-item, two-tiered list of KDEs focusing mainly on urinary and rectal symptoms. The Expanded Prostate Cancer Index Composite (EPIC-26) questionnaire was considered most robust for PROM multi-domain monitoring, and granular KDEs were defined for DCM. CONCLUSIONS This expert consensus on PCa-specific KDEs has served as a foundation for a professional society-endorsed, publicly available operational ontology developed by the American Association of Physicists in Medicine (AAPM) Big Data Sub Committee (BDSC).
Collapse
Affiliation(s)
- Amy Moreno
- Department of Radiation Oncology, University of Texas MD Anderson Cancer Center, Houston, TX 77030, USA
| | - Abhishek A Solanki
- Department of Radiation Oncology, Loyola University Medical Center, Berwyn, IL 60402, USA
| | - Tianlin Xu
- Department of Biostatistics, University of Texas MD Anderson Cancer Center, Houston, TX 77030, USA
| | - Ruitao Lin
- Department of Biostatistics, University of Texas MD Anderson Cancer Center, Houston, TX 77030, USA
| | - Jatinder Palta
- Department of Medical Physics, Virginia Commonwealth University, Richmond, VA 23284, USA
| | - Emily Daugherty
- Department of Radiation Oncology, University of Cincinnati College of Medicine, Cincinnati, OH 45267, USA
| | - David Hong
- Department of Radiation Oncology, University of Southern California, Los Angeles, CA 90089, USA
| | - Julian Hong
- Department of Radiation Oncology, University of California San Francisco, San Francisco, CA 93701, USA
| | - Sophia C Kamran
- Department of Radiation Oncology, Massachusetts General Hospital, Boston, MA 02129, USA
| | - Evangelia Katsoulakis
- Department of Radiation Oncology, James A Haley VA Medical Center, Tampa, FL 33612, USA
| | - Kristy Brock
- Department of Imaging Physics, University of Texas MD Anderson Cancer Center, Houston, TX 77030, USA
| | - Mary Feng
- Department of Radiation Oncology, University of California San Francisco, San Francisco, CA 93701, USA
| | - Clifton Fuller
- Department of Radiation Oncology, University of Texas MD Anderson Cancer Center, Houston, TX 77030, USA
| | - Charles Mayo
- Department of Radiation Physics, University of Michigan, Ann Arbor, MI 48109, USA
| |
Collapse
|
11
|
Caston NE, Franks JA, Balas N, Eltoum N, Thigpen H, Patterson M, Azuero A, Ojesina AI, Dent DN, Hildreth K, Lalor FR, McGowen C, Huang CHS, Dionne-Odom JN, Weiner BJ, Jackson BE, Basch EM, Stover AM, Howell D, Pierce JY, Rocque GB. Evaluating Nurses' Time to Response by Severity and Cancer Stage in a Remote Symptom Monitoring Program for Patients With Breast Cancer. JCO Clin Cancer Inform 2023; 7:e2300015. [PMID: 37279409 PMCID: PMC10530733 DOI: 10.1200/cci.23.00015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Revised: 03/10/2023] [Accepted: 04/13/2023] [Indexed: 06/08/2023] Open
Abstract
PURPOSE Remote symptom monitoring (RSM) using electronic patient-reported outcomes enables patients with cancer to communicate symptoms between in-person visits. A better understanding of key RSM implementation outcomes is crucial to optimize efficiency and guide implementation efforts. This analysis evaluated the association between the severity of patient-reported symptom alerts and time to response by the health care team. METHODS This secondary analysis included women with stage I-IV breast cancer who received care at a large academic medical center in the Southeastern United States (October 2020-September 2022). Symptom surveys with at least one severe symptom alert were categorized as severe. Response time was categorized as optimal if the alert was closed by a health care team member within 48 hours. Odds ratios (ORs), predicted probabilities, and 95% CIs were estimated using a patient-nested logistic regression model. RESULTS Of 178 patients with breast cancer included in this analysis, 63% of patients identified as White and 85% of patients had a stage I-III or early-stage cancer. The median age at diagnosis was 55 years (IQR, 42-65). Of 1,087 surveys included, 36% reported at least one severe symptom alert and 77% had an optimal response time by the health care team. When compared with surveys that had no severe symptom alerts, surveys with at least one severe symptom alert had similar odds of having an optimal response time (OR, 0.97; 95% CI, 0.68 to 1.38). The results were similar when stratified by cancer stage. CONCLUSION Response times to symptom alerts were similar for alerts with at least one severe symptom compared with alerts with no severe symptoms. This suggests that alert management is being incorporated into routine workflows and not prioritized based on disease or symptom alert severity.
Collapse
Affiliation(s)
- Nicole E. Caston
- Department of Medicine, Division of Hematology and Oncology, University of Alabama at Birmingham, Birmingham, AL
| | - Jeffrey A. Franks
- Department of Medicine, Division of Hematology and Oncology, University of Alabama at Birmingham, Birmingham, AL
| | - Nora Balas
- Institute for Cancer Outcomes and Survivorship, University of Alabama at Birmingham, Birmingham, AL
| | - Noon Eltoum
- Department of Medicine, Division of Hematology and Oncology, University of Alabama at Birmingham, Birmingham, AL
| | - Haley Thigpen
- Department of Medicine, Division of Hematology and Oncology, University of Alabama at Birmingham, Birmingham, AL
| | - Megan Patterson
- Department of Medicine, Division of Hematology and Oncology, University of Alabama at Birmingham, Birmingham, AL
| | - Andres Azuero
- O'Neal Comprehensive Cancer Center, Birmingham, AL
- School of Nursing, University of Alabama at Birmingham, Birmingham, AL
| | - Akinyemi I. Ojesina
- Department of Obstetrics and Gynecology, Medical College of Wisconsin, Milwaukee, WI
| | - D'Ambra N. Dent
- Department of Medicine, Division of Hematology and Oncology, University of Alabama at Birmingham, Birmingham, AL
| | - Keyonsis Hildreth
- Department of Medicine, Division of Hematology and Oncology, University of Alabama at Birmingham, Birmingham, AL
| | - Fallon R. Lalor
- Department of Medicine, Division of Hematology and Oncology, University of Alabama at Birmingham, Birmingham, AL
| | - Chelsea McGowen
- University of South Alabama Mitchell Cancer Institute, Mobile, AL
| | - Chao-Hui S. Huang
- Department of Medicine, Division of Gerontology, Geriatrics, and Palliative Care, University of Alabama at Birmingham, Birmingham, AL
| | - J. Nicholas Dionne-Odom
- School of Nursing, University of Alabama at Birmingham, Birmingham, AL
- Department of Medicine, Division of Gerontology, Geriatrics, and Palliative Care, University of Alabama at Birmingham, Birmingham, AL
| | - Bryan J. Weiner
- Department of Health Systems and Population Health, University of Washington, Seattle, WA
| | - Bradford E. Jackson
- Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Ethan M. Basch
- Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Angela M. Stover
- Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill, Chapel Hill, NC
- Department of Health Policy and Management, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Doris Howell
- Supportive Care, Princess Margaret Cancer Centre Research Institute, Toronto, ON, Canada
| | | | - Gabrielle B. Rocque
- Department of Medicine, Division of Hematology and Oncology, University of Alabama at Birmingham, Birmingham, AL
- O'Neal Comprehensive Cancer Center, Birmingham, AL
| |
Collapse
|
12
|
Feng G, Parthipan M, Breunis H, Puts M, Emmenegger U, Timilshina N, Hansen AR, Finelli A, Krzyzanowska MK, Matthew A, Clarke H, Mina DS, Soto-Perez-de-Celis E, Tomlinson G, Alibhai SMH. Feasibility and acceptability of remote symptom monitoring (RSM) in older adults during treatment for metastatic prostate cancer. J Geriatr Oncol 2023; 14:101469. [PMID: 36917921 DOI: 10.1016/j.jgo.2023.101469] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2022] [Revised: 02/20/2023] [Accepted: 02/28/2023] [Indexed: 03/14/2023]
Abstract
INTRODUCTION Emerging data support multiple benefits of remote symptom monitoring (RSM) during chemotherapy to improve outcomes. However, these studies have not focused on older adults and do not include treatments beyond chemotherapy. Although chemotherapy, androgen receptor axis-targeted therapies (ARATs), and radium-223 prolong survival, toxicities are substantial and increased in older adults with metastatic prostate cancer (mPC). We aimed to assess RSM feasibility among older adults receiving life-prolonging mPC treatments. MATERIALS AND METHODS Older adults aged 65+ starting chemotherapy, an ARAT, or radium-223 for mPC were enrolled in a multicentre prospective cohort study. As part of the RSM package, participants completed the Edmonton Symptom Assessment Scale (ESAS) daily and detailed questionnaires assessing mood, anxiety, fatigue, insomnia, and pain weekly online or by phone throughout one treatment cycle (3-4 weeks). Alerts were sent to the clinical oncology team for severe symptoms (ESAS ≥7). Participants also completed an end of study questionnaire that assessed study burden and satisfaction. Descriptive statistics were used to determine recruitment and retention rates, participant response rates to daily and weekly questionnaires, clinician responses to alerts, and participant satisfaction rates. An inductive descriptive approach was used to categorize open-ended responses about study benefits, challenges, and recommendations into relevant themes. RESULTS Ninety males were included (mean age 77 years, 48% ARAT, 38% chemotherapy, and 14% radium-223). Approximately 38% of patients preferred phone-based RSM. Patients provided RSM responses in 1216 out of 1311 daily questionnaires (93%). Over 93% of participants were satisfied (36%), very satisfied (43%), or extremely satisfied (16%) with RSM, although daily reporting was reported by several (8%) as burdensome. Nearly 45% of patients reported severe symptoms during RSM. Most symptom alerts sent to the oncology care team were acknowledged (97%) and 53% led to follow-ups with a nurse or physician for additional care. DISCUSSION RSM is feasible and acceptable to older adults with mPC, but accommodation needs to be made for phone-based RSM. The optimal frequency and duration of RSM also needs to be established.
Collapse
Affiliation(s)
- Gregory Feng
- Department of Medicine, University Health Network, Toronto, Canada.
| | | | | | - Martine Puts
- Lawrence S. Bloomberg Faculty of Nursing, University of Toronto, Toronto, Canada.
| | - Urban Emmenegger
- Division of Medical Oncology, Odette Cancer Centre, Toronto, Canada.
| | | | - Aaron R Hansen
- Division of Medical Oncology and Hematology, Princess Margaret Cancer Centre, Toronto, Canada.
| | - Antonio Finelli
- Division of Urology, Princess Margaret Cancer Centre, Toronto, Canada.
| | - Monika K Krzyzanowska
- Division of Medical Oncology and Hematology, Princess Margaret Cancer Centre, Toronto, Canada.
| | - Andrew Matthew
- Department of Supportive Care, Princess Margaret Cancer Centre, Toronto, Canada.
| | - Hance Clarke
- Department of Anesthesia, Toronto General Hospital, Toronto, Canada.
| | - Daniel Santa Mina
- Faculty of Kinesiology and Physical Education, University of Toronto, Toronto, Canada.
| | - Enrique Soto-Perez-de-Celis
- Department of Geriatrics, Salvador Zubirán National Institute of Medical Sciences and Nutrition, Mexico City, Mexico.
| | - George Tomlinson
- Biostatistics Research Unit, University Health Network, Toronto, Canada.
| | - Shabbir M H Alibhai
- Department of Medicine, University Health Network, Toronto, Canada; Department of Supportive Care, Princess Margaret Cancer Centre, Toronto, Canada.
| |
Collapse
|
13
|
Basch E, Rocque G, Mody G, Mullangi S, Patt D. Tenets for Implementing Electronic Patient-Reported Outcomes for Remote Symptom Monitoring During Cancer Treatment. JCO Clin Cancer Inform 2023; 7:e2200187. [PMID: 36857630 DOI: 10.1200/cci.22.00187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2022] [Revised: 01/09/2023] [Accepted: 01/17/2023] [Indexed: 03/03/2023] Open
Abstract
PURPOSE Symptoms are common in patients receiving systemic treatment for metastatic cancer. Monitoring patients with electronic patient-reported outcomes (ePROs) detects severe and worsening symptoms early, enabling care teams to intervene and prevent downstream complications and thereby improving outcomes. The Centers for Medicare & Medicaid Services will require patient-reported outcome (PRO) monitoring in the upcoming Enhancing Oncology Model, and many practices will likely attempt to implement PROs in patient care for the first time. METHODS To assist practices with the design and implementation of ePRO remote symptom monitoring programs, tenets were drawn from prior ePRO program experiences and research. RESULTS Successful implementation requires a quality improvement approach to change management with attention to software functionality, measured outcomes, personnel deployment, leadership and culture, workflow, equity, and patient engagement. Specific approaches in each of these areas can optimize program participation and effectiveness. Continuous program monitoring to identify and address barriers is essential to success. Initial challenges with personnel acceptance and patient participation are common and can be overcome by using these tenets. CONCLUSION Remote symptom monitoring with ePROs is a key component of quality cancer care and population health management that requires organizational commitment and a deliberate approach by practices using established tenets to assure successful implementation.
Collapse
Affiliation(s)
- Ethan Basch
- University of North Carolina, Chapel Hill, NC
| | | | - Gita Mody
- University of North Carolina, Chapel Hill, NC
| | | | | |
Collapse
|
14
|
Rincones O, Bamgboje-Ayodele A, Arnold A, Delaney GP, Durcinoska I, Avery S, Sandell T, Della-Fiorentina SA, Pearson J, Girgis A. Cancer Care Team's Management of Clinical Alerts Generated by Electronically Collected Patient Reported Outcomes: We Could Do Better. Int J Environ Res Public Health 2023; 20:2001. [PMID: 36767367 PMCID: PMC9915302 DOI: 10.3390/ijerph20032001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/30/2022] [Revised: 01/15/2023] [Accepted: 01/17/2023] [Indexed: 06/18/2023]
Abstract
Electronically administered patient-reported outcome measures (ePROMs) are effective digital health tools for informing clinicians about cancer patients' symptoms and facilitating timely patient-centred care. This paper describes the delivery of healthcare activities supported by the PROMPT-Care model, including ePROMs generated clinical alerts, cancer care team (CCT) response to alerts, and patients' perceptions of the CCT response and ePROMs system. This mixed-methods study includes cancer patients from four cancer therapy centres in New South Wales, Australia. Quantitative and qualitative data were collected regarding clinical alert activity, CCT response, and patient perceptions of the CCT responses and ePROMs system. Qualitative data were thematically analysed. Of the 328 participants whose care was informed by the digital health tool, 70.8% (n = 233) generated at least one alert during the trial period, with 877 alerts generated in total. Although 43.7% (n = 383) were actioned by the CCT, at least 80% of participants found follow-up CCT phone calls beneficial, with multiple benefits confirmed in interviews. The cancer care delivery arm of the PROMPT-Care trial involving clinical alerts to the CCT was positively perceived by most participants, resulting in a diverse range of benefits. However, further work is required, informed by implementation science, to improve the percentage of actioned clinical alerts.
Collapse
Affiliation(s)
- Orlando Rincones
- Ingham Institute for Applied Medical Research, South West Sydney Clinical Campuses, UNSW Medicine & Health, University of New South Wales, Liverpool, NSW 2170, Australia
| | - Adeola Bamgboje-Ayodele
- Ingham Institute for Applied Medical Research, South West Sydney Clinical Campuses, UNSW Medicine & Health, University of New South Wales, Liverpool, NSW 2170, Australia
- Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, Sydney, NSW 2006, Australia
| | - Anthony Arnold
- Wollongong Hospital, Illawarra Shoalhaven Local Health District, Wollongong, NSW 2500, Australia
| | - Geoff P. Delaney
- Ingham Institute for Applied Medical Research, South West Sydney Clinical Campuses, UNSW Medicine & Health, University of New South Wales, Liverpool, NSW 2170, Australia
- Liverpool Cancer Therapy Centre, South Western Sydney Local Health District, Liverpool, NSW 2170, Australia
| | - Ivana Durcinoska
- Ingham Institute for Applied Medical Research, South West Sydney Clinical Campuses, UNSW Medicine & Health, University of New South Wales, Liverpool, NSW 2170, Australia
| | - Sandra Avery
- Liverpool Cancer Therapy Centre, South Western Sydney Local Health District, Liverpool, NSW 2170, Australia
| | - Tiffany Sandell
- Wollongong Hospital, Illawarra Shoalhaven Local Health District, Wollongong, NSW 2500, Australia
| | - Stephen A. Della-Fiorentina
- School of Medicine, Western Sydney University, Campbelltown, NSW 2560, Australia
- Macarthur Cancer Therapy Centre, Campbelltown Hospital, Campbelltown, NSW 2116, Australia
| | - Joanne Pearson
- Liverpool Cancer Therapy Centre, South Western Sydney Local Health District, Liverpool, NSW 2170, Australia
| | - The PROMPT-Care Co-Authorship Group
- Ingham Institute for Applied Medical Research, South West Sydney Clinical Campuses, UNSW Medicine & Health, University of New South Wales, Liverpool, NSW 2170, Australia
- Wollongong Hospital, Illawarra Shoalhaven Local Health District, Wollongong, NSW 2500, Australia
- Liverpool Cancer Therapy Centre, South Western Sydney Local Health District, Liverpool, NSW 2170, Australia
- Macarthur Cancer Therapy Centre, Campbelltown Hospital, Campbelltown, NSW 2116, Australia
| | - Afaf Girgis
- Ingham Institute for Applied Medical Research, South West Sydney Clinical Campuses, UNSW Medicine & Health, University of New South Wales, Liverpool, NSW 2170, Australia
| |
Collapse
|