An Educational Assessment of Evidence Used for Variant Classification: A Report of the Association for Molecular Pathology

Multi-Institutional Evaluation of Interrater Agreement of Biomarker-Drug Pair Rankings Based on the ESMO Scale for Clinical Actionability of molecular Targets (ESCAT) and Sources of Discordance

BACKGROUND

The widespread use of next-generation sequencing in clinical practice has contributed to the accumulation of a large number of genomic findings associated with targeted therapy; therefore, the problem of ranking the detected findings has become acute. The European Society for Medical Oncology Scale of Clinical Actionability of molecular Targets (ESCAT) system was designed by the European Society for Medical Oncology to rank biomarkers into levels of evidence that reflect their potency and clinical significance based on published clinical data. However, the ESCAT system remains imperfect, as it is based on a subjective assessment of the levels of evidence.

OBJECTIVE

The objective of this study was to determine whether the ranking of LOE for biomarker-drug pairs based on the ESCAT system is dependent on the human factor, and to uncover potential issues associated with the use of the framework.

METHODS

To evaluate the inter-rater agreement, we created a dataset of a total of 154 biomarker-drug pairs for 18 unique tumor types. We aimed to include biomarker-drug pairs that could be considered standard of care as well as less common and under investigated pairs. Fourteen precision oncology experts were invited to assign an ESCAT level of evidence for biomarker-drug pairs. Statistical analysis was carried out using Cohen's kappa and the Kolmogorov-Smirnov test.

RESULTS

The inter-rater agreement was low with some exceptions, and significant deviations from the consensus level of evidence were observed. For biomarker-drug associations, the deviations from the consensus were observed for more than 50% of the contributors' rankings. The most agreement between the contributors was observed for lung adenocarcinoma (p < 0.005), while the most disagreement was observed for esophageal cancer (p < 0.01) biomarker-drug pairs in our dataset.

CONCLUSIONS

This study demonstrates noteworthy discordances between the precision oncology experts and may provide the directions for future developments in modifying the ESCAT framework and the overall applicability of the results of genomic profiling into clinical practice.

Comparison of the ABC and ACMG systems for variant classification

The ABC and ACMG variant classification systems were compared by asking mainly European clinical laboratories to classify variants in 10 challenging cases using both systems, and to state if the variant in question would be reported as a relevant result or not as a measure of clinical utility. In contrast to the ABC system, the ACMG system was not made to guide variant reporting but to determine the likelihood of pathogenicity. Nevertheless, this comparison is justified since the ACMG class determines variant reporting in many laboratories. Forty-three laboratories participated in the survey. In seven cases, the classification system used did not influence the reporting likelihood when variants labeled as "maybe report" after ACMG-based classification were included. In three cases of population frequent but disease-associated variants, there was a difference in favor of reporting after ABC classification. A possible reason is that ABC step C (standard variant comments) allows a variant to be reported in one clinical setting but not another, e.g., based on Bayesian-based likelihood calculation of clinical relevance. Finally, the selection of ACMG criteria was compared between 36 laboratories. When excluding criteria used by less than four laboratories (<10%), the average concordance rate was 46%. Taken together, ABC-based classification is more clear-cut than ACMG-based classification since molecular and clinical information is handled separately, and variant reporting can be adapted to the clinical question and phenotype. Furthermore, variants do not get a clinically inappropriate label, like pathogenic when not pathogenic in a clinical context, or variant of unknown significance when the significance is known.

Observations from a national sample exchange program for molecular haematology testing

External quality assessment programs (EQAP) for molecular haematology generally only assess the analytical phase of laboratory testing or provide limited evaluation of post-analytical components. We incorporated comprehensive post-analytical evaluation into an existing national inter-laboratory sample exchange program for molecular haematology due to the increasing complexity of diagnostic molecular testing and interpretation. We report key findings from four years of longitudinal data using this approach. Eighteen participating laboratories enrolled in an annual reciprocal sample exchange program from 2019-2022, which covered conventional and next-generation sequencing (NGS) assays. Participants submitted results on their laboratory information system-generated reports which then underwent central review. Reports were assessed according to consensus values and relevant national and international reporting standards and guidelines. A total of 680 reports were received. Laboratories had high concordance in the analytical phase of testing, with incorrect variant detection observed in a total of six of 680 (0.9%) reports. In contrast, post-analytical concordance was much lower, with at least one discordance observed in 28.9-57.6% of all conventional reports and 33.3-100% NGS reports. The most frequent post-analytical discordances were: (1) not including key technical information on reports (total 41.9% conventional, 47.2% NGS); (2) not using standard gene and variant nomenclature (total 28.2% conventional, 25.6% NGS). NGS reports also demonstrated discrepancies in variant classification (total 20.4%) and interpretation (total 10.2%). The rate of discrepancies generally improved year-on-year. Inter-laboratory concordance for molecular haematology testing is high in the analytical phase, however opportunities exist for improvement in the post-analytical phase. Given that result interpretation is crucial for clinical decision-making and that molecular testing is a complex and evolving field, we suggest that EQAPs should comprehensively evaluate both analytical and post-analytical components of laboratory performance in order to harmonise reporting and to support the accurate interpretation of molecular haematology tests.