Evaluation of an online arts-based platform to support the health and well-being of older adults during the COVID-19 pandemic: a cross-sectional survey

OBJECTIVES

The objective of this study was to conduct a formative evaluation of the Art Your Service (AYS) arts-based program to determine the program's potential for improving the social and physical well-being of older adults during the COVID-19 pandemic.

DESIGN, SETTINGS AND PARTICIPANTS

An online questionnaire was administered to the AYS members who consented to be invited to participate in the study. Questionnaire items consisted of a Likert scale and open-ended questions delivered using an online platform (SurveyMonkey). Participants provided feedback on their perceptions and experiences of the AYS program, such as its impact on their health and well-being during the COVID-19 pandemic, the benefits and challenges of participating, and any suggestions for program improvement.

OUTCOME MEASURES AND ANALYSIS

Quantitative data were analyzed using descriptive statistics (frequencies, means with standard deviations), and open-ended questions (qualitative data) were analyzed using content analysis. Outcomes included participant demographics, perceptions about the program, usability (System Usability Scale [SUS]), eHealth literacy (eHealth Literacy Scale), and social isolation (Lubben Social Network Scale; LSNS-6).

RESULTS

Program participants revealed consistent patterns of their perceptions and experiences about the program, including a high satisfaction rate (95%) and a perceived positive impact on participants' health and well-being. The program sessions were perceived to be a well-organized, convenient, and safe way to engage with one another socially during the COVID-19 pandemic. The program usability was also perceived to be high (SUS mean score 86.2). Participants felt a sense of connectedness and had reduced feelings of social isolation. Most participants (75%) reported that the program improved their physical health.

CONCLUSIONS

Findings from this formative evaluation study identified key strengths and opportunities to improve the Art Your Service arts-based program, which can be used to help enhance the program's functioning and long-term sustainability potential.

A scoping review reveals candidate quality indicators of knowledge translation and implementation science practice tools

OBJECTIVES

To identify candidate quality indicators from existing tools that provide guidance on how to practice knowledge translation and implemenation science (KT practice tools) across KT domains (dissemination, implementation, sustainability, and scalability).

STUDY DESIGN AND SETTING

We conducted a scoping review using the Joanna Briggs Institute Manual for Evidence Synthesis. We systematically searched multiple electronic databases and the gray literature. Documents were independently screened, selected, and extracted by pairs of reviewers. Data about the included articles, KT practice tools, and candidate quality indicators were analyzed, categorized, and summarized descriptively.

RESULTS

Of 43,060 titles and abstracts that were screened from electronic databases and gray literature, 850 potentially relevant full-text articles were identified, and 253 articles were included in the scoping review. Of these, we identified 232 unique KT practice tools from which 27 unique candidate quality indicators were generated. The identified candidate quality indicators were categorized according to the development (n = 17), evaluation (n = 5) and adaptation (n = 3) of the tools, and engagement of knowledge users (n = 2). No tools were identified that appraised the quality of KT practice tools.

CONCLUSIONS

The development of a quality appraisal instrument of KT practice tools is needed. The results will be further refined and finalized in order to develop a quality appraisal instrument for KT practice tools.

Occupational Therapist Led Cognitive Stimulation Therapy: Feasibility of Implementation

Background. Despite local training opportunities for Cognitive Stimulation Therapy (CST) for occupational therapists, there has been limited evaluation of the feasibility of implementation in Canada. Purpose. This study explored the feasibility of CST delivery by an occupational therapist in an Ontario health care setting and the impact it may have on self-efficacy and hope measures of individuals with dementia. Methods. A mixed-methods experimental pre/post design was used. Survey measures included hope, self-efficacy, feasibility, satisfaction, and individual session evaluation. Semistructured focus groups were held for participants and facilitators. Findings. Quantitative findings are summarized descriptively for the 10 participants. Qualitative findings were grouped into themes: social connectedness, knowledge gained and shared, tailored implementation adjustments, and need for long-term programing. Implications. Occupational therapists are well-positioned to implement CST and should be a part of further research to test the intervention rigorously for applicability within a range of Canadian health care settings.

Identifying candidate quality indicators of tools that support the practice of knowledge translation: a scoping review protocol

OBJECTIVE

The objective of this scoping review is to identify and characterize relevant knowledge translation methods tools (those that provide guidance for optimized knowledge translation practice) to uncover candidate quality indicators to inform a future quality assessment tool for knowledge translation strategies.

INTRODUCTION

Knowledge translation strategies (defined as including knowledge translation interventions, tools, and products) target various knowledge users, including patients, clinicians, researchers, and policy-makers. The development and use of strategies that support knowledge translation practice have been rapidly increasing, making it difficult for knowledge users to decide which to use. There is limited evidence-based guidance or measures to help assess the overall quality of knowledge translation strategies.

INCLUSION CRITERIA

Empirical and non-empirical documents will be considered if they explicitly describe a knowledge translation methods tool and its development, evaluation or validation, methodological strengths or limitations, and/or use over time. The review will consider a knowledge translation methods tool if it falls within at least one knowledge translation domain (ie, implementation, dissemination, sustainability, scalability, integrated knowledge translation) in the health field.

METHODS

We will conduct a systematic search of relevant electronic databases and gray literature. The search strategy will be developed iteratively by an experienced medical information specialist and peer-reviewed with the PRESS checklist. The search will be limited to English-only documents published from 2005 onward. Documents will be independently screened, selected, and extracted by 2 researchers. Data will be analyzed and summarized descriptively, including the characteristics of the included documents, knowledge translation methods tools, and candidate quality indicators.

SCOPING REVIEW REGISTRATION

Open Science Framework ( https://osf.io/chxvq ).

Training the Next Generation of Diabetes Researchers: Evaluation of the Diabetes Action Canada Training and Mentoring Program

BACKGROUND

The Diabetes Action Canada Training and Mentoring (DAC-TM) Program launched in June 2017, with the goal of building capacity in the next generation of diabetes researchers in Canada in patient-oriented research (POR).

METHODS

We conducted a program evaluation of the DAC-TM program using a sequential, mixed-methods research design.

RESULTS

Our analysis of 82 surveys and 22 in-depth interviews from a wide range of DAC-TM Program stakeholders revealed consistent patterns in experience with the program. The training sessions were perceived to be well-organized, convenient educational opportunities to gain new knowledge about POR and become integrated into a community of practice of POR researchers who study diabetes and its complications in Canada. The content of the training was perceived to be useful and relevant to participants, although improvements could be made to help address the training needs of the broader DAC community. There is broad support for and appreciation of the mentorship awards, which were perceived to be appropriately targeted to early-career investigators. The mentor-mentee relationships were perceived to be positive, productive and career-advancing overall, but could benefit from a more strategic design and promote better connectivity to foster mentor-mentee relationships. In addition, feedback about opportunities to network and forge new connections was mixed and represents another opportunity for improvement to strengthen capacity building.

CONCLUSIONS

Findings from this formative evaluation study show key strengths and opportunities to improve the DAC-TM Program, which can be used to help enhance its function and promote its long-term sustainability.

Choosing Wisely: An idea worth sustaining

OBJECTIVES

To evaluate the sustainability potential of Choosing Wisely (CW) to address unnecessary medical care at Ontario community hospitals.

DATA SOURCES/STUDY SETTING

Ontario community hospitals and their affiliated family health teams (FHTs).

STUDY DESIGN

A mixed-methods study involving the administration of a validated sustainability survey to CW implementation teams followed by their participation in focus groups.

DATA COLLECTION/EXTRACTION METHODS

Survey data were collected using an Excel file with an embedded, automated scoring system. We collated individual survey scores and generated aggregate team scores. We also performed descriptive statistics for quantitative data (frequencies, means). Qualitative data were triangulated with quantitative assessments to support data interpretations using the meta-matrix method.

PRINCIPAL FINDINGS

Fifteen CW implementation teams across four Ontario community hospitals and six affiliated primary care FHTs participated. CW priority areas investigated were de-prescribing of proton pump inhibitors (PPIs) and reducing Pre-Op testing and BUN/Urea lab testing. Survey results showed steady improvements in sustainability scores from baseline to final follow-up among most implementation teams: 10% increase for PPI de-prescribing (six FHTs) and 2% increase (three hospital teams); 18% increase in BUN/Urea lab testing (three hospital teams). Regardless of site or CW priority area, common facilitators were fit with existing processes and workflows, leadership support, and optimized team communication; common challenges were lack of awareness and buy-in, leadership engagement or a champion, and lack of fit with existing workflow and culture. All teams identified at least one challenge for which they co-designed and implemented a plan to maximize the sustainability potential of their CW initiative.

CONCLUSIONS

Evaluating the sustainability potential of an innovation such as Choosing Wisely is critical to ensuring that they have the best potential for impact. Our work highlights that implementation teams can be empowered to influence implementation efforts and to realize positive outcomes for their health care services and patients.

The ASH-ASPHO Choosing Wisely Campaign: 5 hematologic tests and treatments to question

Choosing Wisely is a medical stewardship and quality-improvement initiative led by the American Board of Internal Medicine Foundation in collaboration with leading medical societies in the United States. The American Society of Hematology (ASH) has been an active participant in the Choosing Wisely project. In 2019, ASH and the American Society of Pediatric Hematology/Oncology (ASPHO) formed a joint task force to solicit, evaluate, and select items for a pediatric-focused Choosing Wisely list. By using an iterative process and an evidence-based method, the ASH-ASPHO Task Force identified 5 hematologic tests and treatments that health care providers and patients should question because they are not supported by evidence, and/or they involve risks of medical and financial costs with low likelihood of benefit. The ASH-ASPHO Choosing Wisely recommendations are as follows: (1) avoid routine preoperative hemostatic testing in an otherwise healthy child with no previous personal or family history of bleeding, (2) avoid platelet transfusion in asymptomatic children with a platelet count 10 × 10³ /μL unless an invasive procedure is planned, (3) avoid thrombophilia testing in children with venous access-associated thrombosis and no positive family history, (4) avoid packed red blood cells transfusion for asymptomatic children with iron deficiency anemia and no active bleeding, and (5) avoid routine administration of granulocyte colony-stimulating factor for prophylaxis of children with asymptomatic autoimmune neutropenia and no history of recurrent or severe infections. We recommend that health care providers carefully consider the anticipated risks and benefits of these identified tests and treatments before performing them.

Effectiveness of an eHealth self-management tool for older adults with multimorbidity (KeepWell): protocol for a hybrid effectiveness-implementation randomised controlled trial

INTRODUCTION

In response to the burden of chronic disease among older adults, different chronic disease self-management tools have been created to optimise disease management. However, these seldom consider all aspects of disease management are not usually developed specifically for seniors or created for sustained use and are primarily focused on a single disease. We created an eHealth self-management application called 'KeepWell' that supports seniors with complex care needs in their homes. It incorporates the care for two or more chronic conditions from among the most prevalent high-burden chronic diseases.

METHODS AND ANALYSIS

We will evaluate the effectiveness, cost and uptake of KeepWell in a 6-month, pragmatic, hybrid effectiveness-implementation randomised controlled trial. Older adults age ≥65 years with one or more chronic conditions who are English speaking are able to consent and have access to a computer or tablet device, internet and an email address will be eligible. All consenting participants will be randomly assigned to KeepWell or control. The allocation sequence will be determined using a random number generator.Primary outcome is perceived self-efficacy at 6 months. Secondary outcomes include quality of life, health background/status, lifestyle (nutrition, physical activity, caffeine, alcohol, smoking and bladder health), social engagement and connections, eHealth literacy; all collected via a Health Risk Questionnaire embedded within KeepWell (intervention) or a survey platform (control). Implementation outcomes will include reach, effectiveness, adoption, fidelity, implementation cost and sustainability.

ETHICS AND DISSEMINATION

Ethics approval has been received from the North York General Hospital Research and Ethics Board. The study is funded by the Canadian Institutes of Health Research and the Ontario Ministry of Health. We will work with our team to develop a dissemination strategy which will include publications, presentations, plain language summaries and an end-of-grant meeting.

TRIAL REGISTRATION NUMBER

NCT04437238.

Improving KT tools and products: development and evaluation of a framework for creating optimized, Knowledge-activated Tools (KaT)

Background

Positive impacts of quality improvement initiatives on health care and services have not been substantial. Knowledge translation (KT) strategies (tools, products and interventions) strive to facilitate the uptake of knowledge thereby the potential to improve care, but there is little guidance on how to develop them. Existing KT guidance or planning tools fall short in operationalizing all aspects of KT practice activities conducted by knowledge users (researchers, clinicians, patients, decision-makers), and most do not consider their variable needs or to deliver recommendations that are most relevant and useful for them.

Methods

We conducted a 3-phase study. In phase 1, we used several sources to develop a conceptual framework for creating optimized Knowledge-activated Tools (KaT) (consultation with our integrated KT team, the use of existing KT models and frameworks, findings of a systematic review of multimorbidity interventions and a literature review and document analysis on existing KT guidance tools). In phase 2, we invited KT experts to participate in a Delphi study to refine and evaluate the conceptual KaT framework. In phase 3, we administered an online survey to knowledge users (researchers, clinicians, decision-makers, trainees) to evaluate the potential usefulness of an online mock-up version of the KaT framework.

Results

We developed the conceptual KaT framework, and iteratively refined it with 35 KT experts in a 3-round Delphi study. The final framework represents the blueprint for what is needed to create KT strategies. Feedback from 201 researcher, clinician, decision-maker and trainee knowledge users on the potential need and usefulness of an online, interactive version of KaT indicated that they liked the idea of it (mean score 4.36 on a 5-point Likert scale) and its proposed features (mean score range 4.30-4.79).

Conclusions

Our findings suggest that mostly Canadian KT experts and knowledge users perceived the KaT framework and the future development of an online, interactive version to be important and needed. We anticipate that the KaT framework will provide clarity for knowledge users about how to identify their KT needs and what activities can address these needs, and to help streamline the process of these activities to facilitate efficient uptake of knowledge.

Underlying mechanisms of complex interventions addressing the care of older adults with multimorbidity: a realist review

OBJECTIVES

To understand how and why effective multi-chronic disease management interventions influence health outcomes in older adults 65 years of age or older.

DESIGN

A realist review.

DATA SOURCES

Electronic databases including Medline and Embase (inception to December 2017); and the grey literature.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We considered any studies (ie, experimental quasi-experimental, observational, qualitative and mixed-methods studies) as long as they provided data to explain our programme theories and effectiveness review (published elsewhere) findings. The population of interest was older adults (age ≥65 years) with two or more chronic conditions.

ANALYSIS

We used the Realist And MEta-narrative Evidence Syntheses: Evolving Standards (RAMESES) quality and publication criteria for our synthesis aimed at refining our programme theories such that they contained multiple context-mechanism-outcome configurations describing the ways different mechanisms fire to generate outcomes. We created a 3-step synthesis process grounded in meta-ethnography to separate units of data from articles, and to derive explanatory statements across them.

RESULTS

106 articles contributed to the analysis. We refined our programme theories to explain multimorbidity management in older adults: (1) care coordination interventions with the best potential for impact are team-based strategies, disease management programmes and case management; (2) optimised disease prioritisation involves ensuring that clinician work with patients to identify what symptoms are problematic and why, and to explore options that are acceptable to both clinicians and patients and (3) optimised patient self-management is dependent on patients' capacity for selfcare and to what extent, and establishing what patients need to enable selfcare.

CONCLUSIONS

To optimise care, both clinical management and patient self-management need to be considered from multiple perspectives (patient, provider and system). To mitigate the complexities of multimorbidity management, patients focus on reducing symptoms and preserving quality of life while providers focus on the condition that most threaten morbidity and mortality.

PROSPERO REGISTRATION NUMBER

CRD42014014489.

Understanding optimal approaches to patient and caregiver engagement in the development of cancer practice guidelines: a mixed methods study

Background

Practice guidelines (PGs) can assist health care practitioners and patients to make decisions about health care options. A key component of high quality PGs is the consideration of patient values and preferences. A mixed methods study was conducted to understand optimal approaches to patient engagement in the development of cancer PGs.

Methods

Cancer patients, survivors, family members and caregivers were recruited from cancer clinics, follow-up clinics, community support programs, a provincial patient and family advisory committee, and a provincial cancer PG development program. Participants attended a workshop, completed a survey, or participated in a telephone interview, to provide information about PG awareness, attitudes, information needs, training, engagement approaches and barriers and facilitators.

Results

Forty-one participants (12 workshop attendees, 21 survey respondents and 8 interviewees) provided data. For those with no PG development experience, fewer than half were previously aware of PGs but perceived several benefits to the inclusion of this perspective. Common barriers to participation across the groups were time commitment, duration of the PG development process, and financial costs. Positive beliefs about the contributions that could be made and practical considerations (e.g., orientation and training, defined roles and expectations) were identified as key features in the successful integration of patients into the PG development process. There was no single model of engagement favored over another.

Conclusions

Study results align with similar studies in other contexts and with international patient engagement efforts. Findings are being used to test new patient engagement models in a programmatic PG development initiative in Ontario, Canada.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-017-2107-5) contains supplementary material, which is available to authorized users.

The Guideline Implementability Decision Excellence Model (GUIDE-M): a mixed methods approach to create an international resource to advance the practice guideline field

Background

Practice guideline (PG) implementability refers to PG features that promote their use. While there are tools and resources to promote PG implementability, none are based on an evidence-informed and multidisciplinary perspective. Our objectives were to (i) create a comprehensive and evidence-informed model of PG implementability, (ii) seek support for the model from the international PG community, (iii) map existing implementability tools on to the model, (iv) prioritize areas for further investigation, and (v) describe how the model can be used by PG developers, users, and researchers.

Methods

A mixed methods approach was used. Using our completed realist review of the literature of seven different disciplines as the foundation, an iterative consensus process was used to create the beta version of the model. This was followed by (i) a survey of international stakeholders (guideline developers and users) to gather feedback and to refine the model, (ii) a content analysis comparing the model to existing PG tools, and (iii) a strategy to prioritize areas of the model for further research by members of the research team.

Results

The Guideline Implementability for Decision Excellence Model (GUIDE-M) is comprised of 3 core tactics, 7 domains, 9 subdomains, 44 attributes, and 40 subattributes and elements. Feedback on the beta version was received from 248 stakeholders from 34 countries. The model was rated as logical, relevant, and appropriate. Seven PG tools were selected and compared to the GUIDE-M: very few tools targeted the Contextualization and Deliberations domain. Also, fewer of the tools addressed PG appraisal than PG development and reporting functions. These findings informed the research priorities identified by the team.

Conclusions

The GUIDE-M provides an evidence-informed international and multidisciplinary conceptualization of PG implementability. The model can be used by PG developers to help them create more implementable recommendations, by clinicians and other users to help them be better consumers of PGs, and by the research community to identify priorities for further investigation.

Clinical practice guidelines and consensus statements in oncology--an assessment of their methodological quality

BACKGROUND

Consensus statements and clinical practice guidelines are widely available for enhancing the care of cancer patients. Despite subtle differences in their definition and purpose, these terms are often used interchangeably. We systematically assessed the methodological quality of consensus statements and clinical practice guidelines published in three commonly read, geographically diverse, cancer-specific journals. Methods Consensus statements and clinical practice guidelines published between January 2005 and September 2013 in Current Oncology, European Journal of Cancer and Journal of Clinical Oncology were evaluated. Each publication was assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) rigour of development and editorial independence domains. For assessment of transparency of document development, 7 additional items were taken from the Institute of Medicine's standards for practice guidelines and the Journal of Clinical Oncology guidelines for authors of guidance documents.

METHODS

Consensus statements and clinical practice guidelines published between January 2005 and September 2013 in Current Oncology, European Journal of Cancer and Journal of Clinical Oncology were evaluated. Each publication was assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) rigour of development and editorial independence domains. For assessment of transparency of document development, 7 additional items were taken from the Institute of Medicine's standards for practice guidelines and the Journal of Clinical Oncology guidelines for authors of guidance documents.

FINDINGS

Thirty-four consensus statements and 67 clinical practice guidelines were evaluated. The rigour of development score for consensus statements over the three journals was 32% lower than that of clinical practice guidelines. The editorial independence score was 15% lower for consensus statements than clinical practice guidelines. One journal scored consistently lower than the others over both domains. No journals adhered to all the items related to the transparency of document development. One journal's consensus statements endorsed a product made by the sponsoring pharmaceutical company in 64% of cases.

CONCLUSION

Guidance documents are an essential part of oncology care and should be subjected to a rigorous and validated development process. Consensus statements had lower methodological quality than clinical practice guidelines using AGREE II. At a minimum, journals should ensure that that all consensus statements and clinical practice guidelines adhere to AGREE II criteria. Journals should consider explicitly requiring guidelines to declare pharmaceutical company sponsorship and to identify the sponsor's product to enhance transparency.

The AGREE Enterprise: a decade of advancing clinical practice guidelines

BACKGROUND

The original AGREE (Appraisal of Guidelines for REsearch and Evaluation) Instrument was published in 2003, and its revision, the AGREE II, in 2009. Together, they filled an important gap in the guideline and quality of care fields. Ten years later, the AGREE Enterprise reflects on a trajectory of projects and international collaboration that have contributed to advancing the science and quality of practice guidelines and the uptake of AGREE/AGREE II.

FINDINGS

The AGREE Enterprise has undertaken activities to improve the tool and to develop resources to support its use. Since 2003, the uptake and adoption of AGREE by the international community has been swift and broad. A total of 33 language translations of the original AGREE Instrument and the current AGREE II are available and were initiated by the international community. A recent scan of the published literature identified over 600 articles that referenced the AGREE tools. The AGREE tools have been widely received and applied, with several organizations having incorporated the AGREE as part of their formal practice guideline programs. Since its redevelopment in 2010, the AGREE Enterprise website (www.agreetrust.org) continues to experience steady increases in visitors per month and currently has over 10,000 registered users.

CONCLUSIONS

The AGREE Enterprise has contributed to the advancements of guidelines through research activities and international participation by scientific and user communities. As we enter a new decade, we look forward to ongoing collaborations and contributing to further advancements to improve quality of care and health care systems.

A mixed methods approach to understand variation in lung cancer practice and the role of guidelines

INTRODUCTION

Practice pattern data demonstrate regional variation and lower than expected rates of adherence to practice guideline (PG) recommendations for the treatment of stage II/IIIA resected and stage IIIA/IIIB unresected non-small cell lung cancer (NSCLC) patients in Ontario, Canada. This study sought to understand how clinical decisions are made for the treatment of these patients and the role of PGs.

METHODS

Surveys and key informant interviews were undertaken with clinicians and administrators.

RESULTS

Participants reported favorable ratings for PGs and the evidentiary bases underpinning them. The majority of participants agreed more patients should have received treatment and that regional variation is problematic. Participants estimated that up to 30% of patients are not good candidates for treatment and up to 20% of patients refuse treatment. The most common barrier to implementing PGs was the lack of organizational support by clinical administrative leadership. There was concern that the trial results underpinning the PG recommendations were not generalizable to the typical patients seen in clinic. The qualitative analysis yielded five themes related to physicians' decision making: the unique patient, the unique physician, the family, the clinical team, and the clinical evidence. A dynamic interplay between these factors exists.

CONCLUSION

Our study demonstrates the challenges inherent in (i) the complexity of clinical decision making; (ii) how quality of care problems are perceived and operationalized; and (iii) the clinical appropriateness and utility of PG recommendations. We argue that systematic and rigorous methodologies to help decision makers mitigate or negotiate these challenges are warranted.

Making sense of complex data: a mapping process for analyzing findings of a realist review on guideline implementability

BACKGROUND

Realist reviews offer a rigorous method to analyze heterogeneous data emerging from multiple disciplines as a means to develop new concepts, understand the relationships between them, and identify the evidentiary base underpinning them. However, emerging synthesis methods such as the Realist Review are not well operationalized and may be difficult for the novice researcher to grasp. The objective of this paper is to describe the development of an analytic process to organize and synthesize data from a realist review.

METHODS

Clinical practice guidelines have had an inconsistent and modest impact on clinical practice, which may in part be due to limitations in their design. This study illustrates the development of a transparent method for organizing and analyzing a complex data set informed by a Realist Review on guideline implementability to better understand the characteristics of guidelines that affect their uptake in practice (e.g., clarity, format). The data organization method consisted of 4 levels of refinement: 1) extraction and 2) organization of data; 3) creation of a conceptual map of guideline implementability; and 4) the development of a codebook of definitions.

RESULTS

This new method is comprised of four steps: data extraction, data organization, development of a conceptual map, and operationalization vis-a-vis a codebook. Applying this method, we extracted 1736 guideline attributes from 278 articles into a consensus-based set of categories, and collapsed them into 5 core conceptual domains for our guideline implementability map: Language, Format, Rigor of development, Feasibility, Decision-making.

CONCLUSIONS

This study advances analysis methods by offering a systematic approach to analyzing complex data sets where the goals are to condense, organize and identify relationships.

Does a quality-of-care problem exist? Cancer Care Ontario practice pattern data and the recommendations of two lung cancer practice guidelines

190

Background: Practice-pattern data, evidence-based knowledge and transfer, and performance management strategies define Cancer Care Ontario’s quality improvement strategy. Knowledge products, such as practice guidelines, are intended to provide recommendations for practice, based on best available evidence, to improve quality of care and reduce variation in practice. Review of 2010-2011 Cancer System Quality Index (CSQI) data revealed complex practice patterns in treatment of non-small cell lung cancer patients with stages II and IIIa resected and stages IIIa and IIIb non-resected disease in Ontario. A multi-method study was initiated to understand the patterns, to identify if a quality of care problem exists and to propose improvements moving forward. Methods: Surgeons, medical oncologists and radiation oncologists from Ontario were invited to participate in a survey consisting of 6 areas of inquiry. A grounded theory approach was used to guide key informant interviews of purposively sampled clinicians and administrators. A more in-depth analysis of the CSQI data was planned. Results: Clinicians responding to survey provided positive assessments of PG recommendations and evidentiary base; perceptions of practice patterns were less problematic than hypothesized; estimates of benchmarks were highly variable; and assessments of barriers to recommendation implementation included slow referral process, lack of organization support and patients seen in practice not reflected in the evidence. From the interviews, 5 themes emerged: unique patient, unique physician, family, clinical team, and clinical evidence. Further analysis of CSQI data was not possible given limitations related to data collection. Conclusions: A perceived quality of care problem initiated this study. Concerns centred on significant proportion of patients receiving no treatment; modest percentage of patients receiving treatment that aligned with PG recommendations; and regional variation within each of the clinical care options. Our data show that defining a quality of care problem is significantly more complex than consideration of practice patterns alone.

The landscape of knowledge translation interventions in cancer control: what do we know and where to next? A review of systematic reviews

Background

Effective implementation strategies are needed to optimize advancements in the fields of cancer diagnosis, treatment, survivorship, and end-of-life care. We conducted a review of systematic reviews to better understand the evidentiary base of implementation strategies in cancer control.

Methods

Using three databases, we conducted a search and identified English-language systematic reviews published between 2005 and 2010 that targeted consumer, professional, organizational, regulatory, or financial interventions, tested exclusively or partially in a cancer context (primary focus); generic or non-cancer-specific reviews were also considered. Data were extracted, appraised, and analyzed by members of the research team, and research ideas to advance the field were proposed.

Results

Thirty-four systematic reviews providing 41 summaries of evidence on 19 unique interventions comprised the evidence base. AMSTAR quality ratings ranged between 2 and 10. Team members rated most of the interventions as promising and in need of further research, and 64 research ideas were identified.

Conclusions

While many interventions show promise of effectiveness in the cancer-control context, few reviews were able to conclude definitively in favor of or against a specific intervention. We discuss the complexity of implementation research and offer suggestions to advance the science in this area.

E-learning interventions are comparable to user's manual in a randomized trial of training strategies for the AGREE II

Background

Practice guidelines (PGs) are systematically developed statements intended to assist in patient and practitioner decisions. The AGREE II is the revised tool for PG development, reporting, and evaluation, comprised of 23 items, two global rating scores, and a new User's Manual. In this study, we sought to develop, execute, and evaluate the impact of two internet interventions designed to accelerate the capacity of stakeholders to use the AGREE II.

Methods

Participants were randomized to one of three training conditions. 'Tutorial'--participants proceeded through the online tutorial with a virtual coach and reviewed a PDF copy of the AGREE II. 'Tutorial + Practice Exercise'--in addition to the Tutorial, participants also appraised a 'practice' PG. For the practice PG appraisal, participants received feedback on how their scores compared to expert norms and formative feedback if scores fell outside the predefined range. 'AGREE II User's Manual PDF (control condition)'--participants reviewed a PDF copy of the AGREE II only. All participants evaluated a test PG using the AGREE II. Outcomes of interest were learners' performance, satisfaction, self-efficacy, mental effort, time-on-task, and perceptions of AGREE II.

Results

No differences emerged between training conditions on any of the outcome measures.

Conclusions

We believe these results can be explained by better than anticipated performance of the AGREE II PDF materials (control condition) or the participants' level of health methodology and PG experience rather than the failure of the online training interventions. Some data suggest the online tools may be useful for trainees new to this field; however, this requires further study.

Improving cancer control in Canada one case at a time: the "Knowledge Translation in Cancer" casebook

PURPOSE

In-the-field projects aiming to improve quality in cancer control provide a valuable complement to health services and knowledge translation (kt) research studies. The present paper describes the methods used to develop the Knowledge Translation for Cancer Control in Canada: A Casebook and its results.

METHODS

Nominations for in-the-field projects were accepted from individuals and organizations across Canada. The nominations had to demonstrate that a specific cancer control problem was identified; that a deliberate and organized effort was developed and implemented to address the identified problem; and that an evaluation-formal or informal-was used to assess the effort. A selection of nominated cases were chosen for more comprehensive analysis.

RESULTS

Thirty nominations were received. Most tackled problems related to treatment or diagnosis. Challenges related to breast, gastrointestinal, and genitourinary cancer were most common among the disease-specific projects, and most projects were regional in scope, with strategies targeting organizational solutions. Of the 30 nominations, 19 were chosen for further analysis. Of those 19, 5 were influenced by a kt model or theory, and 16 reported formal evaluation strategies. Surveys were the most common evaluation method, and process outcomes and clinical surrogate outcomes were the most frequently cited. Financial and administrative challenges were most often cited as key barriers. The key lessons learned concerned the need for a collaborative high-functioning team, project management, and support.

CONCLUSIONS

The casebook provides tangible examples of in-the-field efforts to improve cancer control and provides practical direction for other individuals and institutions facing similar challenges. We discuss the interface between field projects and research projects in the kt arena and how mutual learning can help to optimize the value of each approach.

Enhancing the effectiveness of clearance for physical activity participation: background and overall process1This paper is one of a selection of papers published in the Special Issue entitled Evidence-based risk assessment and recommendations for physical activity clearance, and has undergone the Journal's usual peer-review process

Recent feedback from physical activity (PA) participants, fitness professionals, and physicians has indicated that there are limitations to the utility and effectiveness of the existing PAR-Q and PARmed-X screening tools for PA participation. The aim of this study was to have authorities in exercise and chronic disease management to work with an expert panel to increase the effectiveness of clearance for PA participation using an evidence-based consensus approach and the well-established Appraisal of Guidelines for Research and Evaluation (AGREE) Instrument. Systematic reviews were conducted to develop a new PA clearance protocol involving risk stratification and a decision-tree process. Evidence-based support was sought for enabling qualified exercise professionals to have a direct role in the PA participation clearance process. The PAR-Q+ was developed to use formalized probes to clarify problematic responses and to explore issues arising from currently diagnosed chronic disease or condition. The original PARmed-X tool is replaced with an interactive computer program (ePARmed-X+) to clear prospective PA participants for either unrestricted or supervised PA or to direct them to obtain medical clearance. Evidence-based validation was also provided for the direct role of highly qualified university-educated exercise professionals in the PA clearance process. The risks associated with exercise during pregnancy were also evaluated. The systematic review and consensus process, conforming to the AGREE Instrument, has provided a sound evidence base for enhanced effectiveness of the clearance process for PA participation of both asymptomatic populations and persons with chronic diseases or conditions.

Development of the AGREE II, part 1: performance, usefulness and areas for improvement

BACKGROUND

We undertook research to improve the AGREE instrument, a tool used to evaluate guidelines. We tested a new seven-point scale, evaluated the usefulness of the original items in the instrument, investigated evidence to support shorter, tailored versions of the tool, and identified areas for improvement.

METHOD

We report on one component of a larger study that used a mixed design with four factors (user type, clinical topic, guideline and condition). For the analysis reported in this article, we asked participants to read a guideline and use the AGREE items to evaluate it based on a seven-point scale, to complete three outcome measures related to adoption of the guideline, and to provide feedback on the instrument's usefulness and how to improve it.

RESULTS

Guideline developers gave lower-quality ratings than did clinicians or policy-makers. Five of six domains were significant predictors of participants' outcome measures (p < 0.05). All domains and items were rated as useful by stakeholders (mean scores > 4.0) with no significant differences by user type (p > 0.05). Internal consistency ranged between 0.64 and 0.89. Inter-rater reliability was satisfactory. We received feedback on how to improve the instrument.

INTERPRETATION

Quality ratings of the AGREE domains were significant predictors of outcome measures associated with guideline adoption: guideline endorsements, overall intentions to use guidelines, and overall quality of guidelines. All AGREE items were assessed as useful in determining whether a participant would use a guideline. No clusters of items were found more useful by some users than others. The measurement properties of the seven-point scale were promising. These data contributed to the refinements and release of the AGREE II.

Development of the AGREE II, part 2: assessment of validity of items and tools to support application

BACKGROUND

We established a program of research to improve the development, reporting and evaluation of practice guidelines. We assessed the construct validity of the items and user's manual in the beta version of the AGREE II.

METHODS

We designed guideline excerpts reflecting high-and low-quality guideline content for 21 of the 23 items in the tool. We designed two study packages so that one low-quality and one high-quality version of each item were randomly assigned to each package. We randomly assigned 30 participants to one of the two packages. Participants reviewed and rated the guideline content according to the instructions of the user's manual and completed a survey assessing the manual.

RESULTS

In all cases, content designed to be of high quality was rated higher than low-quality content; in 18 of 21 cases, the differences were significant (p < 0.05). The manual was rated by participants as appropriate, easy to use, and helpful in differentiating guidelines of varying quality, with all scores above the mid-point of the seven-point scale. Considerable feedback was offered on how the items and manual of the beta-AGREE II could be improved.

INTERPRETATION

The validity of the items was established and the user's manual was rated as highly useful by users. We used these results and those of our study presented in part 1 to modify the items and user's manual. We recommend AGREE II (available at www.agreetrust.org) as the revised standard for guideline development, reporting and evaluation.

Recruitment of multiple stakeholders to health services research: lessons from the front lines

BACKGROUND

Self-administered surveys are an essential methodological tool for health services and knowledge translation research, and engaging end-users of the research is critical. However, few documented accounts of the efforts invested in recruitment of multiple different stakeholders to one health services research study exist. Here, we highlight the challenges of recruiting key stakeholders (policy-makers, clinicians, guideline developers) to a Canadian Institutes of Health Research (CIHR) funded health services research (HSR) study aimed to develop an updated and refined version of a guideline appraisal tool, the AGREE.

METHODS

Using evidence-based methods of recruitment, our goal was to recruit 192 individuals: 80 international guideline developers, 80 Canadian clinicians and 32 Canadian policy/decision-makers. We calculated the participation rate and the recruitment efficiency.

RESULTS

We mailed 873 invitation letters. Of 838 approached, our participation rate was 29%(240) and recruitment efficiency, 19%(156). One policy-maker manager did not allow policy staff to participate in the study.

CONCLUSIONS

Based on the results from this study, we suggest that future studies aiming to engage similar stakeholders in HSR over sample by at least 5 times to achieve their target sample size and allow for participant withdrawals. We need continued efforts to communicate the value of research between researchers and end-users of research (policy-makers, clinicians, and other researchers), integration of participatory research strategies, and promotion of the value of end-user involvement in research. Future research to understand methods of improving recruitment efficiency and engaging key stakeholders in HSR is warranted.

A randomized trial to evaluate e-learning interventions designed to improve learner's performance, satisfaction, and self-efficacy with the AGREE II

BACKGROUND

Practice guidelines (PGs) are systematically developed statements intended to assist in patient, practitioner, and policy decisions. The AGREE II is the revised and updated standard tool for guideline development, reporting and evaluation. It is comprised of 23 items and a user's Manual. The AGREE II is ready for use.

OBJECTIVES

To develop, execute, and evaluate the impact of two internet-based educational interventions designed to accelerate the capacity of stakeholders to use the AGREE II: a multimedia didactic tutorial with a virtual coach, and a higher intensity training program including both the didactic tutorial and an interactive practice exercise component.

METHODS

Participants (clinicians, developers, and policy makers) will be randomly assigned to one of three conditions. Condition one, didactic tutorial -- participants will go through the on-line AGREE II tutorial supported by a virtual coach and review of the AGREE II prior to appraising the test PG. Condition two, tutorial + practice -- following the multimedia didactic tutorial with a virtual coach, participants will review the on-line AGREE II independently and use it to appraise a practice PG. Upon entering their AGREE II score for the practice PG, participants will be given immediate feedback on how their score compares to expert norms. If their score falls outside a predefined range, the participant will receive a series of hints to guide the appraisal process. Participants will receive an overall summary of their performance appraising the PG compared to expert norms. Condition three, control arm -- participants will receive a PDF copy of the AGREE II for review and to appraise the test PG on-line. All participants will then rate one of ten test PGs with the AGREE II. The outcomes of interest are learners' performance, satisfaction, self-efficacy, mental effort, and time-on-task; comparisons will be made across each of the test groups.

DISCUSSION

Our research will test innovative educational interventions of various intensities and instructional design to promote the adoption of AGREE II and to identify those strategies that are most effective for training. The results will facilitate international capacity to apply the AGREE II accurately and with confidence and to enhance the overall guideline enterprise.

Practitioners As Experts: The Influence of Practicing Oncologists “in-the-Field” on Evidence-Based Guideline Development

Purpose

Panels of experts are used to develop clinical practice guidelines (CPGs) intended to be used by practitioners “in-the-field.” Therefore, oncologists’ participation in CPG development is an important strategy to promote CPG adoption. The purpose of this study was to evaluate the contributions of oncologists in-the-field to evidence-based CPG development using data from Ontario’s cancer system.

Methods

CPG development in Ontario includes surveys of oncologists’ opinions, using a structured questionnaire, about draft recommendations that were developed from rigorous systematic reviews of evidence prepared by expert panels. Two research assistants reviewed background documents to trace the changes in CPG recommendations from draft to final stage to determine the contribution of oncologists’ input to final recommendations. Changes to recommendations were categorized as either substantive (content or tone) or minor (ideas clarification or edits).

Results

From 2000 to 2003, 43 CPGs were developed. There were 87 changes to draft recommendations for 31 CPGs, of which 40 changes to 19 CPGs could be attributed to survey input from practicing oncologists. Of the 40 changes, 28 (70%) were judged to be substantive.

Conclusion

Despite a rigorous evidence-based process for CPG development, practicing oncologists contribute substantially to the final recommendations approved by the expert panel. It is hypothesized that the responsiveness of expert panels to input from oncologists in-the-field will facilitate adoption of CPGs.