Development of an international glossary for clinical guidelines collaboration

OBJECTIVES

Clinical practice guidelines (CPGs) are often created through collaboration among organizations. The use of inconsistent terminology may cause poor communication and delays. This study aimed to develop a glossary of terms related to collaboration in guideline development.

STUDY DESIGN AND SETTING

A literature review of collaborative guidelines was performed to develop an initial list of terms related to guideline collaboration. The list of terms was presented to the members of the Guideline International Network Guidelines Collaboration Working Group, who provided presumptive definitions for each term and proposed additional terms to be included. The revised list was subsequently reviewed by an international, multidisciplinary panel of expert stakeholders. Recommendations received during this pre-Delphi review were implemented to augment an initial draft glossary. The glossary was then critically evaluated and refined through two rounds of Delphi surveys and a virtual consensus meeting with all panel members as Delphi participants.

RESULTS

Forty-nine experts participated in the pre-Delphi survey, and 44 participated in the two-round Delphi process. Consensus was reached for 37 terms and definitions.

CONCLUSION

Uptake and utilization of this guideline collaboration glossary by key organizations and stakeholder groups may facilitate collaboration among guideline-producing organizations by improving communication, minimizing conflicts, and increasing guideline development efficiency.

An International Needs Assessment Survey of Guideline Developers Demonstrates Variability in Resources and Challenges to Collaboration between Organizations

BACKGROUND

The development of rigorous, high-quality clinical guidelines increases the need for resources and skilled personnel within guideline-producing organizations. While collaboration between organizations provides a unique opportunity to pool resources and save time and effort, the collaboration presents its own unique challenges.

OBJECTIVE

To assess the perceived needs and current challenges of guideline producers worldwide related to guideline development and collaboration efforts.

DESIGN

Survey questions were developed by the Guidelines International Network and the US GRADE Network, pilot-tested among attendees of a guideline development workshop, and disseminated electronically using convenience and snowball sampling methods.

PARTICIPANTS

A total of 171 respondents representing 30 countries and more than 112 unique organizations were included in this analysis.

MAIN MEASURES

The survey included free-response, multiple-choice, and seven-point Likert-scale questions. Questions assessed respondents' perceived value of guidelines, resource availability and needs, guideline development processes, and collaboration efforts of their organization.

KEY RESULTS

Time required to develop high-quality systematic reviews and guidelines was the most relevant need (median=7; IQR=5.5-7). In-house resources to conduct literature searches (median=4; IQR=3-6) and the resources to develop rigorous guidelines rapidly (median=4; IQR=2-5) were perceived as the least available resources. Difficulties reconciling differences in guideline methodology (median=6; IQR=4-7) and the time required to establish collaborative agreements (median=6; IQR=5-6) were the most relevant barriers to collaboration between organizations. Results also indicated a general need for improvement in conflict of interest (COI) disclosure policies.

CONCLUSION

The survey identified organizational challenges in supporting rigorous guideline development, including the time, resources, and personnel required. Connecting guideline developers to existing databases of high-quality systematic reviews and the use of freely available online platforms may facilitate guideline development. Guideline-producing organizations may also consider allocating resources to hiring or training personnel with expertise in systematic review methodologies or utilizing resources more effectively by establishing collaborations with other organizations.

Evaluating the Adoption of Laboratory Practice Guidelines

Context.—

To date, the College of American Pathologists (CAP) has developed 17 laboratory practice guidelines (LPGs) including updates. In 2013, the CAP was awarded a 5-year cooperative agreement grant from the United States Centers for Disease Control and Prevention to increase the effectiveness of LPGs.

Objective.—

To assess the awareness and adoption of 2 CAP LPGs: immunohistochemical (IHC) assay validation and initial workup of acute leukemia.

Design.—

Baseline surveys for each LPG were conducted in 2010 and 2015, respectively. To measure the adoption of guideline recommendations and inform future updates, a follow-up study consisting of surveys, telephone interviews, and focus group sessions was conducted in laboratories that indicated they perform IHC testing. A follow-up study for the acute leukemia LPG is planned.

Results.—

For the IHC Validation LPG, a total of 1624 survey responses, 40 telephone interviews, and discussions with 5 focus group participants were analyzed. The response rate for the aforementioned 3 modalities was 46%, 13%, and 3%, respectively. All modalities indicated most respondents were aware of the LPG and had adopted most or all of its recommendations. Respondents expressed needs for continued communication, increased specificity, and more prescriptive recommendations when the guideline is updated.

Conclusions.—

While data-driven development of evidence-based LPGs requires significant resources, active data collection to identify gaps and assess adoption contributes to improved laboratory testing practices in support of patient care. The CAP identified sustainable modalities to track metrics and developed multiple tools that should improve guideline development, adoption, and implementation. Of these modalities, written or electronic surveys were the most logistically feasible and had the highest response rate.

Analytic Validation of Immunohistochemistry Assays: New Benchmark Data From a Survey of 1085 Laboratories

CONTEXT

- A cooperative agreement between the College of American Pathologists (CAP) and the United States Centers for Disease Control and Prevention was undertaken to measure laboratories' awareness and implementation of an evidence-based laboratory practice guideline (LPG) on immunohistochemical (IHC) validation practices published in 2014.

OBJECTIVE

- To establish new benchmark data on IHC laboratory practices.

DESIGN

- A 2015 survey on IHC assay validation practices was sent to laboratories subscribed to specific CAP proficiency testing programs and to additional nonsubscribing laboratories that perform IHC testing. Specific questions were designed to capture laboratory practices not addressed in a 2010 survey.

RESULTS

- The analysis was based on responses from 1085 laboratories that perform IHC staining. Ninety-six percent (809 of 844) always documented validation of IHC assays. Sixty percent (648 of 1078) had separate procedures for predictive and nonpredictive markers, 42.7% (220 of 515) had procedures for laboratory-developed tests, 50% (349 of 697) had procedures for testing cytologic specimens, and 46.2% (363 of 785) had procedures for testing decalcified specimens. Minimum case numbers were specified by 85.9% (720 of 838) of laboratories for nonpredictive markers and 76% (584 of 768) for predictive markers. Median concordance requirements were 95% for both types. For initial validation, 75.4% (538 of 714) of laboratories adopted the 20-case minimum for nonpredictive markers and 45.9% (266 of 579) adopted the 40-case minimum for predictive markers as outlined in the 2014 LPG. The most common method for validation was correlation with morphology and expected results. Laboratories also reported which assay changes necessitated revalidation and their minimum case requirements.

CONCLUSIONS

- Benchmark data on current IHC validation practices and procedures may help laboratories understand the issues and influence further refinement of LPG recommendations.

Analytic Validation of Immunohistochemical Assays: A Comparison of Laboratory Practices Before and After Introduction of an Evidence-Based Guideline

CONTEXT

- Laboratories must demonstrate analytic validity before any test can be used clinically, but studies have shown inconsistent practices in immunohistochemical assay validation.

OBJECTIVE

- To assess changes in immunohistochemistry analytic validation practices after publication of an evidence-based laboratory practice guideline.

DESIGN

- A survey on current immunohistochemistry assay validation practices and on the awareness and adoption of a recently published guideline was sent to subscribers enrolled in one of 3 relevant College of American Pathologists proficiency testing programs and to additional nonsubscribing laboratories that perform immunohistochemical testing. The results were compared with an earlier survey of validation practices.

RESULTS

- Analysis was based on responses from 1085 laboratories that perform immunohistochemical staining. Of 1057 responses, 65.4% (691) were aware of the guideline recommendations before this survey was sent and 79.9% (550 of 688) of those have already adopted some or all of the recommendations. Compared with the 2010 survey, a significant number of laboratories now have written validation procedures for both predictive and nonpredictive marker assays and specifications for the minimum numbers of cases needed for validation. There was also significant improvement in compliance with validation requirements, with 99% (100 of 102) having validated their most recently introduced predictive marker assay, compared with 74.9% (326 of 435) in 2010. The difficulty in finding validation cases for rare antigens and resource limitations were cited as the biggest challenges in implementing the guideline.

CONCLUSIONS

- Dissemination of the 2014 evidence-based guideline validation practices had a positive impact on laboratory performance; some or all of the recommendations have been adopted by nearly 80% of respondents.

Evaluation of Testing of Acute Leukemia Samples: Survey Result From the College of American Pathologists

CONTEXT

- The classification and prognosis determination in acute leukemia (AL) are complex and it is unclear what testing is being performed in practice.

OBJECTIVE

- To survey physicians describing their current practice of test ordering in the diagnosis of AL.

DESIGN

- In anticipation of a guideline by the College of American Pathologists (CAP) and the American Society for Hematology on laboratory testing needed for the initial workup of AL, a baseline survey was designed by an expert panel from CAP. Members of professional societies were asked to describe their current practice of test ordering.

RESULTS

- Two hundred ninety-four responses were received with 258 respondents analyzed after the first qualifying survey question regarding initial diagnosis of AL. One hundred seventy-six of 249 respondents (70.7%) were board-certified hematopathologists. Flow cytometry and karyotype analysis were routinely performed for acute myeloid leukemia (AML) (99.1% [232 of 234] and 96.2% [225 of 234], respectively) and acute lymphoblastic leukemia (ALL) (98.3% [229 of 233] and 96.6% [225 of 233], respectively). In addition, fluorescence in situ hybridization studies were routinely performed by 81.2% (190 of 234) of respondents for AML and 85.0% (198 of 233) of respondents for ALL; other molecular studies were performed by 78.2% (183) for AML and 54.9% (128) for ALL; immunohistochemistry by 44.9% (105) for AML and 47.6% (111) for ALL; and cytochemistry by 24.8% (58) for AML and 14.2% (33) for ALL.

CONCLUSIONS

- While flow cytometry and karyotyping are routinely reported as being performed for the diagnosis of AL, there is marked variation in the reporting of testing patterns for other genetic studies, immunohistochemistry, and cytochemistry.

Interpretive Diagnostic Error Reduction in Surgical Pathology and Cytology: Guideline From the College of American Pathologists Pathology and Laboratory Quality Center and the Association of Directors of Anatomic and Surgical Pathology

CONTEXT

Additional reviews of diagnostic surgical and cytology cases have been shown to detect diagnostic discrepancies.

OBJECTIVE

To develop, through a systematic review of the literature, recommendations for the review of pathology cases to detect or prevent interpretive diagnostic errors.

DESIGN

The College of American Pathologists Pathology and Laboratory Quality Center in association with the Association of Directors of Anatomic and Surgical Pathology convened an expert panel to develop an evidence-based guideline to help define the role of case reviews in surgical pathology and cytology. A literature search was conducted to gather data on the review of cases in surgical pathology and cytology.

RESULTS

The panel drafted 5 recommendations, with strong agreement from open comment period participants ranging from 87% to 93%. The recommendations are: (1) anatomic pathologists should develop procedures for the review of selected pathology cases to detect disagreements and potential interpretive errors; (2) anatomic pathologists should perform case reviews in a timely manner to avoid having a negative impact on patient care; (3) anatomic pathologists should have documented case review procedures that are relevant to their practice setting; (4) anatomic pathologists should continuously monitor and document the results of case reviews; and (5) if pathology case reviews show poor agreement within a defined case type, anatomic pathologists should take steps to improve agreement.

CONCLUSIONS

Evidence exists that case reviews detect errors; therefore, the expert panel recommends that anatomic pathologists develop procedures for the review of pathology cases to detect disagreements and potential interpretive errors, in order to improve the quality of patient care.

Principles of analytic validation of immunohistochemical assays: Guideline from the College of American Pathologists Pathology and Laboratory Quality Center

CONTEXT

Laboratories must validate all assays before they can be used to test patient specimens, but currently there are no evidence-based guidelines regarding validation of immunohistochemical assays.

OBJECTIVE

To develop recommendations for initial analytic validation and revalidation of immunohistochemical assays.

DESIGN

The College of American Pathologists Pathology and Laboratory Quality Center convened a panel of pathologists and histotechnologists with expertise in immunohistochemistry to develop validation recommendations. A systematic evidence review was conducted to address key questions. Electronic searches identified 1463 publications, of which 126 met inclusion criteria and were extracted. Individual publications were graded for quality, and the key question findings for strength of evidence. Recommendations were derived from strength of evidence, open comment feedback, and expert panel consensus.

RESULTS

Fourteen guideline statements were established to help pathology laboratories comply with validation and revalidation requirements for immunohistochemical assays.

CONCLUSIONS

Laboratories must document successful analytic validation of all immunohistochemical tests before applying to patient specimens. The parameters for cases included in validation sets, including number, expression levels, fixative and processing methods, should take into account intended use and should be sufficient to ensure that the test accurately measures the analyte of interest in specimens tested in that laboratory. Recommendations are also provided for confirming assay performance when there are changes in test methods, reagents, or equipment.

Validating whole slide imaging for diagnostic purposes in pathology: guideline from the College of American Pathologists Pathology and Laboratory Quality Center

CONTEXT

There is increasing interest in using whole slide imaging (WSI) for diagnostic purposes (primary and/or consultation). An important consideration is whether WSI can safely replace conventional light microscopy as the method by which pathologists review histologic sections, cytology slides, and/or hematology slides to render diagnoses. Validation of WSI is crucial to ensure that diagnostic performance based on digitized slides is at least equivalent to that of glass slides and light microscopy. Currently, there are no standard guidelines regarding validation of WSI for diagnostic use.

OBJECTIVE

To recommend validation requirements for WSI systems to be used for diagnostic purposes.

DESIGN

The College of American Pathologists Pathology and Laboratory Quality Center convened a nonvendor panel from North America with expertise in digital pathology to develop these validation recommendations. A literature review was performed in which 767 international publications that met search term requirements were identified. Studies outside the scope of this effort and those related solely to technical elements, education, and image analysis were excluded. A total of 27 publications were graded and underwent data extraction for evidence evaluation. Recommendations were derived from the strength of evidence determined from 23 of these published studies, open comment feedback, and expert panel consensus.

RESULTS

Twelve guideline statements were established to help pathology laboratories validate their own WSI systems intended for clinical use. Validation of the entire WSI system, involving pathologists trained to use the system, should be performed in a manner that emulates the laboratory's actual clinical environment. It is recommended that such a validation study include at least 60 routine cases per application, comparing intraobserver diagnostic concordance between digitized and glass slides viewed at least 2 weeks apart. It is important that the validation process confirm that all material present on a glass slide to be scanned is included in the digital image.

CONCLUSIONS

Validation should demonstrate that the WSI system under review produces acceptable digital slides for diagnostic interpretation. The intention of validating WSI systems is to permit the clinical use of this technology in a manner that does not compromise patient care.

Consensus statement on effective communication of urgent diagnoses and significant, unexpected diagnoses in surgical pathology and cytopathology from the College of American Pathologists and Association of Directors of Anatomic and Surgical Pathology

CONTEXT

Recognizing the difficulty in applying the concept of critical values to anatomic pathology diagnoses, the College of American Pathologists and the Association of Directors of Anatomic and Surgical Pathology have chosen to reevaluate the concept of critical diagnoses.

OBJECTIVE

To promote effective communication of urgent and significant, unexpected diagnoses in surgical pathology and cytology.

DESIGN

A comprehensive literature search was conducted and reviewed by an expert panel.

RESULTS

A policy of effective communication of important results in surgical pathology and cytology is desirable to enhance patient safety and to address multiple regulatory requirements.

CONCLUSIONS

Each institution should create its own policy regarding urgent diagnoses and significant, unexpected diagnoses in anatomic pathology. This policy should be separate from critical results or panic-value policies in clinical pathology, with the expectation of a different time frame for communication. Urgent diagnosis is defined as a medical condition that, in most cases, should be addressed as soon as possible. Significant, unexpected diagnosis is defined as a medical condition that is clinically unusual or unforeseen and should be addressed at some point in the patient's course. Further details of this statement are provided.

Gynecologic Cytology Proficiency Testing Failures: What Have We Learned?: Observations From the College of American Pathologists Gynecologic Cytology Proficiency Testing Program

Context.—In 2006, the first gynecologic cytology proficiency tests were offered by the College of American Pathologists. Four years of data are now available using field-validated slides, including conventional and liquid-based Papanicolaou tests.

Objective.—To characterize the pattern of error types that resulted in initial proficiency-test failure for cytotechnologists, primary screening pathologists, and secondary pathologists (those whose slides are prescreened by cytotechnologists).

Design.—The results of 37 029 initial College of American Pathologists Papanicolaou proficiency tests were reviewed from 4 slide-set modules: conventional, ThinPrep, SurePath, or a module containing all 3 slide types.

Results.—During this 4-year period, cytotechnologists were least likely to fail the initial test (3.4%; 614 of 18 264), followed by secondary pathologists (ie, those reviewing slides already screened by a cytotechnologist) with a failure rate of 4.2% (728 of 17 346), and primary pathologists (ie, those screening their own slides) having the highest level of failure (13.7%; 194 of 1419). Failure rates have fallen for all 3 groups over time. Pathologists are graded more stringently on proficiency tests, and more primary pathologists would have passed if they had been graded as cytotechnologists. There were no significant differences among performances using different types of slide sets. False-positive errors were common for both primary (63.9%; 124 of 194 errors) and secondary (55.6%; 405 of 728 errors) pathologists, whereas automatic failures were most common for cytotechnologists (75.7%; 465 of 614 errors).

Conclusions.—The failure rate is decreasing for all participants. The failures for primary pathologist screeners are due to false-positive responses. Primary screening cytotechnologists and secondary pathologists have automatic failures more often than do primary screening pathologists.

Granulomatous inflammation-an underestimated cause of false-positive diagnoses in lung fine-needle aspirates: observations from the college of american pathologists nongynecologic cytopathology interlaboratory comparison program

CONTEXT

The false-positive rate for fine-needle aspirates of the lung has been cited as less than 1% for granulomatous inflammation, comprising one of the known causes of false-positive diagnoses.

OBJECTIVE

To determine the rate of false-positive diagnoses of granulomatous inflammation for lung fine-needle aspirates by assessing the false-positive response rate in the context of the College of American Pathologists Nongynecologic Cytopathology Interlaboratory Comparison Program.

DESIGN

We performed a retrospective review of 1092 participant responses for lung fine-needle aspirate challenges with the reference diagnosis of specific infections/granulomatous inflammation from 1998 to 2008 from the College of American Pathologists Nongynecologic Cytopathology Interlaboratory Comparison Program. False-positive rates by participant type (pathologist versus cytotechnologist), general diagnosis category, reference diagnosis, and preparation type were analyzed for the pathologists' responses.

RESULTS

Of the 502 general category responses for pathologists, 428 (85.3%) were benign, 55 (11%) were malignant, and 19 (3.8%) were suspicious. There was no difference in the false-positive rate between preparations (P  =  .76) or participants (P  =  .39). Of those responses by pathologists that were benign, only 68.7% (292 of 425) were an exact match to granulomatous inflammation. Non-small cell carcinoma, adenocarcinoma, and squamous carcinoma represented 64% of false-positive/suspicious responses, while small cell carcinoma and carcinoid comprised 13%.

CONCLUSION

In an interlaboratory comparison program, granulomatous inflammation represents an important cause of false-positive/suspicious responses in lung fine-needle aspirates (14.8%) and is much higher than false-positive rates reported historically in clinical studies. These results highlight the importance of granulomatous inflammation as a mimic of carcinoma.

Practices of participants in the college of american pathologists interlaboratory comparison program in cervicovaginal cytology, 2006

CONTEXT

Liquid-based preparations (LBPs) and human papillomavirus testing have led to changes in cervical cytology practices. The College of American Pathologists attempts to track practice patterns using a supplemental questionnaire, which allows laboratories to report diagnostic practices.

OBJECTIVE

To analyze the 2006 reporting practices and to compare the results with the 2003 survey data.

DESIGN

Questionnaire was mailed to 1621 laboratories. Participants included laboratories enrolled in the 2006 College of American Pathologists Gynecologic Proficiency Testing Program or the educational Interlaboratory Comparison Program in Gynecologic Cytology.

RESULTS

Of the 679 responding laboratories (response rate, 42%), most (97.8%; n = 664) had implemented the Bethesda 2001 terminology. The median rate for all preparations with low-grade squamous intraepithelial lesions was 2.5% (2.9% for LBPs) compared with a 2003 median rate of 2.1%; the increase was confined to LBPs. Rates for high-grade squamous intraepithelial lesions (median, 0.5%) and atypical squamous cells have changed little. High-grade squamous intraepithelial lesions and unsatisfactory rates varied at statistically significant levels between types of LBPs. Most atypical squamous cell cases were subclassified as undetermined significance (median, 4.3%). The median ratio of atypical squamous cells to squamous intraepithelial lesions and carcinomas for all specimen types combined was 1.5, similar to the 2003 median ratio of 1.4. The median rates for findings of squamous cell abnormalities for 2006 were significantly higher for LBPs than for conventional smears.

CONCLUSIONS

Most responding laboratories have implemented the Bethesda 2001 terminology. There is an increase in LBP low-grade squamous intraepithelial lesion rates when compared with 2003 data. Liquid-based preparations have higher median squamous intraepithelial lesion and atypical squamous cell rates.

Unsatisfactory reporting rates: 2006 practices of participants in the college of american pathologists interlaboratory comparison program in gynecologic cytology

CONTEXT

-Minimum cellular criteria for satisfactory Papanicolaou tests were established with the Bethesda System in 2001, and unsatisfactory rates are used as a quality-reporting measure.

OBJECTIVE

-To evaluate practices and unsatisfactory rates from laboratories responding to the 2007 College of American Pathologists supplemental questionnaire survey.

DESIGN

-In 2007, a supplemental questionnaire was mailed to 1621 laboratories enrolled in the 2006 College of American Pathologists Interlaboratory Comparison Program in Gynecologic Cytology (PAP Education), requesting data from the 2006 calendar year. Unsatisfactory rates, reasons for unsatisfactory specimens, laboratory size, and specimen preparation type were analyzed.

RESULTS

-A total of 42% of the laboratories responded to the survey. Most of those laboratories (637 of 674; 94.5%) used the Bethesda System minimum cellularity criteria. Of those laboratories responding, 79% (527 of 667) used the Bethesda System criteria for atrophic or postirradiation specimens. Unsatisfactory rates have increased since 1996. SurePath preparations were associated with the lowest unsatisfactory rate (50th percentile, 0.30; 95th percentile, 1.3), conventional Papanicolaou tests had the highest 95th percentile rates (50th percentile, 1.0; 95th percentile, 5.90), and ThinPrep specimens had the highest median percentile (50th percentile, 1.1; 95th percentile, 3.4). The most-common reason for unsatisfactory Papanicolaou tests was too few squamous cells. Air-drying artifact was the least-common reason for unsatisfactory reporting for liquid-based preparations.

CONCLUSIONS

-Use of the Bethesda System criteria for unsatisfactory specimens is widespread. Unsatisfactory rates have increased since 1996; however, the median rates are 1.1% or less for all preparations. Results from the College of American Pathologists PAP Education supplemental questionnaire continue to provide valuable benchmarking data for cytologic quality-improvement programs in laboratories.

Guidelines for the reporting of nongynecologic cytopathology specimens

CONTEXT

Gynecologic cytology terminology and report formatting have been nationally standardized since the implementation of The Bethesda System of 1988, but standard reporting for nongynecologic cytology has never been formally addressed on the same scale.

OBJECTIVES

To promote patient safety through uniform reporting in nongynecologic cytology (including fine-needle aspiration cytology) and to improve communication between laboratories and health care providers.

DATA SOURCES

Sources include the College of American Pathologists Cytopathology Resource Committee; the College of American Pathologists Council on Scientific Affairs Ad Hoc Committee on Pathology Report Standardization; the College of American Pathologists Laboratory Accreditation Program inspection checklists; the Joint Commission for Accreditation of Healthcare Organizations; and the Clinical Laboratory Improvement Amendments of 1988.

CONCLUSIONS

We describe the major elements of quality nongynecologic cytology reporting and discuss areas of controversy in cytology reporting. Standardized nongynecologic specimen reporting will expand the concept of common report elements already widely implemented in gynecologic cytology reporting. The intent is to improve communication with the health care team while remaining in compliance with federal mandates and accreditation guidelines.

Automatic failure in gynecologic cytology proficiency testing. Results from the College of American Pathologists proficiency testing program

CONTEXT

Automatic failure in gynecologic cytology proficiency testing occurs when a high-grade lesion or carcinoma (HSIL+, Category D) is misinterpreted as negative for intraepithelial lesion or malignancy (Category B).

OBJECTIVES

To document the automatic failure rate in 2006 and 2007 from the College of American Pathologists proficiency testing program (PAP PT) and compare them to projected values from 2004.

DESIGN

Identify automatic failures from PAP PT in 2006 and 2007 and compare the rates of failure regarding participant and preparation type to validated slides in the College of American Pathologists Interlaboratory Comparison Program in 2004.

RESULTS

There were 65 264 participant responses for HSIL+ slides included in this analysis from 2006 and 2007. Overall, 1% (666 of 65 264) of the HSIL+ responses were classified as negative, resulting in automatic failure for the participant. There were significantly fewer automatic failures in 2007 as compared with either 2006 or projected from 2004 data (P < .001). Conventional preparations had a lower automatic failure rate than liquid-based preparations but only for 2006. Both pathologists and cytotechnologists interpreting liquid-based preparations faired better than projected from 2004 data.

CONCLUSIONS

The automatic failure rate in PAP PT is lower than expected based on 2004 data from the College of American Pathologists Interlaboratory Comparison Program. Automatic failures are a relatively small component (1% or less) of proficiency testing failures. The rate of automatic failure decreased from 2006 to 2007 and may be due to loss of poor performers in the testing pool, the test-taking environment, or removal of less robust slides from the program.

Performance of herpes simplex challenges in proficiency testing: observations from the College of American Pathologists proficiency testing program

CONTEXT

Herpes simplex virus (HSV) is a common sexually transmitted disease that is decreasing in prevalence. Herpes simplex virus challenge cases have performed well in the educational College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology. However, since gynecologic cytology proficiency testing (PT) was instituted, performance of HSV challenge cases has worsened.

OBJECTIVE

To determine if the decrease in performance for HSV challenges is due to inexperience in identifying herpes virus or "gaming" of PT.

DESIGN

We compare graded (field-validated) and ungraded (educational) slides with HSV prior to PT (1991- 2005) and after PT (2006 and 2007) in an attempt to characterize changes in performance of slides demonstrating purely HSV cellular features. All HSV slides were negative for intraepithelial lesion with the specific reference interpretation of HSV. Slides were excluded from analysis if they demonstrated more than one diagnostic category or had squamous intraepithelial lesion in addition to HSV changes.

RESULTS

A total of 40 634 HSV responses were evaluated. A significant difference existed in the diagnosis of HSV in PT compared with graded challenges prior to PT (96.6 versus 98.6; P < .001). No difference was seen in educational challenges pre-PT and post-PT (P = .14). No difference was present between slide preparation type; however, a difference was seen between cytotechnologist and pathologist response.

CONCLUSIONS

There is a change in the performance of slide challenges demonstrating the cellular features of HSV in PT, most likely due to testing strategies of participants, rather than lack of recognition of HSV secondary to decreased prevalence.

Performance of Candida--fungal-induced atypia and proficiency testing: observations from the College of American Pathologists proficiency testing program

CONTEXT

Candida may elicit cellular changes on otherwise negative screening Papanicolaou tests that may be misinterpreted as atypical squamous cells of undetermined significance. Although these changes have been correctly interpreted in the educational program of the College of American Pathologists, the Interlaboratory Comparison Program in Gynecologic Cytology, the performance of negative slides with Candida faltered when the same field validated slides were included in proficiency testing (PT).

OBJECTIVE

To identify the performance differences of negative for intraepithelial lesion (NILM) Candida challenges before and after PT.

DESIGN

We compare the performance of NILM College of American Pathologists slides with Candida as a single reference diagnosis, prior to PT (1991-2006) and after PT (2006-2007).

RESULTS

There were 147,186 responses for slides with NILM Candida from the College of American Pathologists programs from 1991 through 2007. After PT, 79.7% of incorrect participant responses identified Candida as Category C (low-grade squamous intraepithelial lesion), whereas prior to PT only 59.5% of the incorrect diagnoses were low-grade squamous intraepithelial lesion (P < .001) in the field-validated component of the program. Validated Candida slides performed significantly more poorly in PT (97.2%) than prior to PT (98.3%) (P < .001). Candida challenges performed better in the educational component post-PT (98.3% versus 97.2%; P < .001). Cytotechnologists (97.9%) identified Candida more frequently than pathologists (97.3%) (P < .001) and ThinPrep preparations performed the best of all preparation types.

CONCLUSIONS

Proficiency testing adversely affects the performance of participants in the identification of NILM Candida. Slides with Candida are more likely to be identified as low-grade squamous intraepithelial lesion in a PT exercise. Misidentification is not due to lack of recognition but most likely an attempt of test takers to optimize their likelihood of passing the examination.

Appeals in gynecologic cytology proficiency testing: review and analysis of data from the 2006 College of American Pathologists gynecologic cytology proficiency testing program

CONTEXT

In 2006, 9643 participants took the initial College of American Pathologists (CAP) Proficiency Test (PT). Failing participants may appeal results on specific test slides. Appeals are granted if 3 referee pathologists do not unanimously agree on the initial reference diagnosis in a masked review process.

OBJECTIVES

To investigate causes of PT failures, subsequent appeals, and appeal successes in 2006.

DESIGN

Appeals were examined, including patient demographic information, Centers for Medicare and Medicaid Services category (A, B, C, or D), exact reference diagnosis, examinees per appeal, examinee's Centers for Medicare and Medicaid Services category, referee's Centers for Medicare and Medicaid Services category, slide preparation type, and slide field validation rate.

RESULTS

There was a 94% passing rate for 2006. One hundred fifty-five examinees (1.6%) appealed 86 slides of all preparation types. Forty-five appeals (29%) were granted on 21 slides; 110 appeals (72%) were denied on 65 slides. Reference category D and B slides were most often appealed. The highest percentage of granted appeals occurred in category D (35% slides; 42% of participants) and the lowest occurred in category B (9% slides; 8% of participants). The field validation rate of all appealed slides was greater than 90%.

CONCLUSIONS

Despite rigorous field validation of slides, 6% of participants failed. Thirty percent of failing participants appealed; most appeals involved misinterpretation of category D as category B. Referees were never unanimous in their agreement with the participant. The participants and referees struggled with the reliability and reproducibility of finding rare cells, "overdiagnosis" of benign changes, and assigning the morphologically dynamic biologic changes of squamous intraepithelial lesions to static categories.

Summary of the 2006 College of American Pathologists Gynecologic Cytology Proficiency Testing Program

CONTEXT

Creating a tool that assesses professional proficiency in gynecologic cytology is challenging. A valid proficiency test (PT) must reflect practice conditions, evaluate locator and interpretive skills, and discriminate between those practitioners who are competent and those who need more education. The College of American Pathologists Gynecologic Cytology Proficiency Testing Program (PAPPT) was approved to enroll participants in a nationwide PT program in 2006.

OBJECTIVE

Report results from the 2006 PAPPT program.

DESIGN

Summarize PT results by pass/fail rate, participant type, and slide-set modules.

RESULTS

Nine thousand sixty-nine participants showed initial PT failure rates of 5%, 16%, and 6% for cytotechnologists, primary screening pathologists, and secondary screening pathologist, respectively. The overall initial test failure rate was 6%. After 3 retests, 9029 (99.6%) of the participants were able to achieve compliance with the PT requirement. No participant "tested out"; however, 40 individuals "dropped out" of the testing sequence (8 cytotechnologists, 9 primary screening pathologists, 23 secondary screening pathologists). Initial failure rates by slide-set modules were 6% conventional, 6% ThinPrep, 6% SurePath, and 5% mixture of all 3 slide types.

CONCLUSIONS

A total of 99.6% of individuals enrolled in the 2006 PAPPT program achieved satisfactory results. The data confirm that cytotechnologists have higher initial pass rates than pathologists and pathologists who are secondary screeners perform better than those who are primary screeners. There was no difference identified in overall pass rates between the slide-set modules. Further analysis of data should help define the results and ongoing challenges of providing a nationwide federally mandated proficiency testing program in gynecologic cytology.