Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory

Development and assessment of autoverification system for routine coagulation assays in inpatient and outpatient settings of tertiary care hospital: algorithm performance and impact on laboratory efficiency

OBJECTIVES

This study aimed to develop and implement autoverification (AV) system for routine coagulation assays, specifically prothrombin time (PT) and activated partial thromboplastin time (APTT), in tertiary care hospital. The efficiency, accuracy, and impact on turnaround time (TAT) were evaluated.

METHODS

AV rules were developed using historical data from 70,865 coagulation test results. The rules included pre-analytical, analytical, and post-analytical checks. The system underwent validation through data simulations, pilot phase, go-live implementation. Performance metrics included sensitivity, specificity, predictive values, passing rates, error rates, TAT.

RESULTS

The AV system achieved 63.3 % overall passing rate (analyzed from 159,183 data), with outpatient settings showing higher rate (69.2 %) than inpatient settings (56.3 %). Final performance evaluation showed sensitivity, specificity, PPV, and NPV of 93.0 , 65.0, 59.7, and 94.4 %, respectively. Manual verification was required for 36 % of cases, mainly due to defective sample volumes (21.5 %). False negatives, primarily from partial clots, occurred in 0.1 % of cases. Integrating CBC clot alerts into AV rules halved the errors. The system increased tests completed within guaranteed TAT of 90 min by 2.4 %, from 89.7 to 92.1 % and reduced median TAT by 5 min. Outpatient TAT improved significantly, with a reduction over 19 min.

CONCLUSIONS

The AV system for APTT and PT tests was successfully implemented, reducing manual verification, improving TAT, particularly in outpatient settings. This study highlights AV systems' potential to enhance laboratory performance for routine coagulation panels, which rely only on APTT and PT assays. Ongoing rule refinement and monitoring remain crucial for enhancing system accuracy and effectiveness.

Establishment and application of autoverification system for HbA1c testing

Introduction

This study aimed to determine autoverification rules for routine glycated hemoglobin (HbA1c) analysis based on high-performance liquid chromatography (HPLC) principle. Laboratory information system (LIS) and Bio-Rad D-100 Advisor software (Bio-Rad, Hercules, USA) with graphics recognition function were carriers for the autoverification system.

Materials and methods

A total of 105,126 HbA1c results, including 98,249 HbA1c matching fast plasma glucose (FPG) results of real-world data from May 2019 to June 2020, were collected to determine autoverification rules including flags, delta checks, reporting limits, and logical rules. The validation database was composed of 48,045 HbA1c results and 41,083 matching FPG results. Autoverification passing rate and the reduction of turnaround time (TAT) were evaluated.

Results

Four autoverification systems (A, B, C, D) were established by two types of delta check rules, 28 flags, one reporting limits, and two kinds of logical rules. The autoverification passing rates were 80.6%, 78.8%, 83.7%, and 81.3%, and the average time saved in TAT were 117.5 min, 116.7 min, 121.1 min, and 121.7 min, respectively.

Conclusions

Autoverification system C was the optimal one. Application of distribution of FPG corresponding to HbA1c groups had better performance as logical rules. Established HbA1c autoverifcation system shortened the auditing report time and improved work efficiency.

Benefit analysis of the auto-verification system of intelligent inspection for microorganisms

In recent years, the automatic machine for microbial identification and antibiotic susceptibility tests has been introduced into the microbiology laboratory of our hospital, but there are still many steps that need manual operation. The purpose of this study was to establish an auto-verification system for bacterial naming to improve the turnaround time (TAT) and reduce the burden on clinical laboratory technologists. After the basic interpretation of the gram staining results of microorganisms, the appearance of strain growth, etc., the 9 rules were formulated by the laboratory technologists specialized in microbiology for auto-verification of bacterial naming. The results showed that among 70,044 reports, the average pass rate of auto-verification was 68.2%, and the reason for the failure of auto-verification was further evaluated. It was found that the main causes reason the inconsistency between identification results and strain appearance rationality, the normal flora in the respiratory tract and urine that was identified, the identification limitation of the mass spectrometer, and so on. The average TAT for the preliminary report of bacterial naming was 35.2 h before, which was reduced to 31.9 h after auto-verification. In summary, after auto-verification, the laboratory could replace nearly 2/3 of manual verification and issuance of reports, reducing the daily workload of medical laboratory technologists by about 2 h. Moreover, the TAT on the preliminary identification report was reduced by 3.3 h on average, which could provide treatment evidence for clinicians in advance.

Automation in the Thrombosis and Hemostasis Laboratory

Automation continues to advance into hemostasis and thrombosis laboratories. Integration of hemostasis testing into an existing chemistry track systems and adoption of a separate hemostasis track systems are important considerations. Unique issues must be addressed to maintain quality and efficiency when automation is introduced. Among other challenges, this chapter discusses centrifugation protocols, incorporation of specimen-check modules in the workflow, and inclusion of tests amenable to automation.

Post-analytical Issues in Hemostasis and Thrombosis Testing: An Update

There are typically three phases identified as contributing to the total testing process. The preanalytical phase starts with the clinician and the patient, when laboratory testing is being considered. This phase also includes decisions about which tests to order (or not), patient identification, blood collection, blood transport, sample processing, and storage to name a few. There are many potential failures that may occur in this preanalytical phase, and these are the topic of another chapter in this book. The second phase, the analytical phase, represents the performance of the test, which is essentially covered in various protocols in this book and the previous edition. The third is the post-analytical phase, which is what occurs after sample testing, and is the topic of the current chapter. Post-analytical issues are generally related to reporting and interpretation of test results. This chapter provides a brief description of these events, as well as guidance for preventing or minimizing post-analytical issues. In particular, there are several strategies for improved post-analytical reporting of hemostasis assays, with this providing the final opportunity to prevent serious clinical errors in patient diagnosis or management.

Designing and validating an autoverification system of biochemical test results in Hatay Mustafa Kemal University, clinical laboratory

Introduction

Autoverification (AV) is a postanalytical tool that uses algorithms to validate test results according to specified criteria. The Clinical and Laboratory Standard Institute (CLSI) document for AV of clinical laboratory test result (AUTO-10A) includes recommendations for laboratories needing guidance on implementation of AV algorithms. The aim was to design and validate the AV algorithm for biochemical tests.

Materials and methods

Criteria were defined according to AUTO-10A. Three different approaches for algorithm were used as result limit checks, which are reference range, reference range ± total allowable error, and 2nd and 98th percentile values. To validate the algorithm, 720 cases in middleware were tested. For actual cases, 3,188,095 results and 194,520 reports in laboratory information system (LIS) were evaluated using the AV system. Cohen’s kappa (κ) was calculated to determine the degree of agreement between seven independent reviewers and the AV system.

Results

The AV passing rate was found between 77% and 85%. The highest rates of AV were in alanine transaminase (ALT), direct bilirubin (DBIL), and magnesium (Mg), which all had AV rates exceeding 85%. The most common reason for non-validated results was the result limit check (41%). A total of 328 reports evaluated by reviewers were compared to AV system. The statistical analysis resulted in a κ value between 0.39 and 0.63 (P < 0.001) and an agreement rate between 79% and 88%.

Conclusions

Our improved model can help laboratories design, build, and validate AV systems and be used as starting point for different test groups.

Customized middleware experience in a tertiary care hospital hematology laboratory

Background

In the clinical laboratory, middleware is a software application that sits between the analyzer and the laboratory information system (LIS). One of the more common uses of middleware is to perform more efficient result autoverification than can be achieved by the LIS or analyzer alone. In addition to autoverification, middleware can support highly customized rules to handle samples and results from specific patient locations. The objective of this study was to review the impact of customized middleware rules that were designed and implemented in the hematology laboratory of a 1000-bed tertiary care adult academic center hospital.

Methods

Three novel initiatives using middleware rules to achieve workflow efficiencies were retrospectively reviewed over different audit periods: preliminary neutrophil resulting for oncology patients, microcytosis interpretive comments, and 1 white blood cell differential (WBCD) reported per day. In addition, autoverification rates for complete blood count and differential (CBCD) and coagulation tests were calculated.

Results

A preliminary neutrophil count was released from middleware on average 64 min before the final CBCD for Leukemia/Bone Marrow Transplant (L/BMT) outpatients, and on average 59 min earlier for oncology patients. Reflexing interpretive comments for select instances of microcytosis removed on average 500 slides per month from technologist review with an estimated cost savings of approximately $3383.33 CAD per month. The 1 WBCD per day rule resulted in a 5.1% cancelation rate, resulting in an estimated monthly cost savings of $943.46 CAD in reagents and technologist time. Finally, middleware rules achieved very high autoverification rates of 97.2% and 88.3% for CBC and CBCD results, respectively.

Conclusions

Implementation of customized middleware hematology rules in our institution resulted in multiple positive impacts on workflow, achieving high autoverification rates, reduced slide reviews, cost savings, and improved standardization.

The development of autoverification system of lymphocyte subset assays on the flow cytometry platform

OBJECTIVES

Peripheral blood lymphocyte subsets are important parameters for monitoring immune status; however, lymphocyte subset detection is time-consuming and error-prone. This study aimed to explore a highly efficient and clinically useful autoverification system for lymphocyte subset assays performed on the flow cytometry platform.

METHODS

A total of 94,402 lymphocyte subset test results were collected. To establish the limited-range rules, 80,427 results were first used (69,135 T lymphocyte subset tests and 11,292 NK, B, T lymphocyte tests), of which 15,000 T lymphocyte subset tests from human immunodeficiency virus (HIV) infected patients were used to set customized limited-range rules for HIV infected patients. Subsequently, 13,975 results were used for historical data validation and online test validation.

RESULTS

Three key autoverification rules were established, including limited-range, delta-check, and logical rules. Guidelines for addressing the issues that trigger these rules were summarized. The historical data during the validation phase showed that the total autoverification passing rate of lymphocyte subset assays was 69.65% (6,941/9,966), with a 67.93% (5,268/7,755) passing rate for T lymphocyte subset tests and 75.67% (1,673/2,211) for NK, B, T lymphocyte tests. For online test validation, the total autoverification passing rate was 75.26% (3,017/4,009), with 73.23% (2,191/2,992) for the T lymphocyte subset test and 81.22% (826/1,017) for the NK, B, T lymphocyte test. The turnaround time (TAT) was reduced from 228 to 167 min using the autoverification system.

CONCLUSIONS

The autoverification system based on the laboratory information system for lymphocyte subset assays reduced TAT and the number of error reports and helped in the identification of abnormal cell populations that may offer clues for clinical interventions.

Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results

Background

Validation of the autoverification function is one of the critical steps to confirm its effectiveness before use. It is crucial to verify whether the programmed algorithm follows the expected logic and produces the expected results. This process has always relied on the assessment of human–machine consistency and is mostly a manually recorded and time-consuming activity with inherent subjectivity and arbitrariness that cannot guarantee a comprehensive, timely and continuous effectiveness evaluation of the autoverification function. To overcome these inherent limitations, we independently developed and implemented a laboratory information system (LIS)-based validation system for autoverification.

Methods

We developed a correctness verification and integrity validation method (hereinafter referred to as the "new method") in the form of a human–machine dialog. The system records personnel review steps and determines whether the human–machine review results are consistent. Laboratory personnel then analyze the reasons for any inconsistency according to system prompts, add to or modify rules, reverify, and finally improve the accuracy of autoverification.

Results

The validation system was successfully established and implemented. For a dataset consisting of 833 rules for 30 assays, 782 rules (93.87%) were successfully verified in the correctness verification phase, and 51 rules were deleted due to execution errors. In the integrity validation phase, 24 projects were easily verified, while the other 6 projects still required the additional rules or changes to the rule settings. Taking the Hepatitis B virus test as an example, from the setting of 65 rules to the automated releasing of 3000 reports, the validation time was reduced from 452 (manual verification) to 275 h (new method), a reduction in validation time of 177 h. Furthermore, 94.6% (168/182) of laboratory users believed the new method greatly reduced the workload, effectively controlled the report risk and felt satisfied. Since 2019, over 3.5 million reports have been automatically reviewed and issued without a single clinical complaint.

Conclusion

To the best of our knowledge, this is the first report to realize autoverification validation as a human–machine interaction. The new method effectively controls the risks of autoverification, shortens time consumption, and improves the efficiency of laboratory verification.

Error Evaluation in the Laboratory Testing Process and Laboratory Information Systems

Introduction: The laboratory testing process consists of five analysis phases featuring the total testing process framework. Activities in laboratory process, including those of testing, are error-prone and affect the use of laboratory information systems. This study seeks to identify error factors related to system use and the first and last phases of the laboratory testing process using a proposed framework known as total testing process-laboratory information systems. Materials and Methods: We conducted a qualitative case study in two private hospitals and a medical laboratory. We collected data using interviews, observations, and document analysis methods involving physicians, nurses, an information technology officer, and the laboratory staff. We employed the proposed framework and Lean problem-solving tools namely Value Stream Mapping and A3 for data analysis. Results: Errors in laboratory information systems and the laboratory testing process were attributed to failure to fulfill user requirements, poor cooperation between the information technology unit and laboratory, the inconsistency of software design in system integration, errors during inter-system data transmission, and lack of motivation in system use. The error factors are related to system development elements, namely, latent failures that considerably affected the information quality and system use. Errors in system development were also attributed to poor service quality. Conclusion: Complex laboratory testing process and laboratory information systems require rigorous evaluation in minimizing errors and ensuring patient safety. The proposed framework and Lean approach are applicable for evaluating the laboratory testing process and laboratory information systems in a rigorous, comprehensive, and structured manner.

Use of Middleware Data to Dissect and Optimize Hematology Autoverification

Background:

Hematology analysis comprises some of the highest volume tests run in clinical laboratories. Autoverification of hematology results using computer-based rules reduces turnaround time for many specimens, while strategically targeting specimen review by technologist or pathologist.

Methods:

Autoverification rules had been developed over a decade at an 800-bed tertiary/quarternary care academic medical central laboratory serving both adult and pediatric populations. In the process of migrating to newer hematology instruments, we analyzed the rates of the autoverification rules/flags most commonly associated with triggering manual review. We were particularly interested in rules that on their own often led to manual review in the absence of other flags. Prior to the study, autoverification rates were 87.8% (out of 16,073 orders) for complete blood count (CBC) if ordered as a panel and 85.8% (out of 1,940 orders) for CBC components ordered individually (not as the panel).

Results:

Detailed analysis of rules/flags that frequently triggered indicated that the immature granulocyte (IG) flag (an instrument parameter) and rules that reflexed platelet by impedance method (PLT-I) to platelet by fluorescent method (PLT-F) represented the two biggest opportunities to increase autoverification. The IG flag threshold had previously been validated at 2%, a setting that resulted in this flag alone preventing autoverification in 6.0% of all samples. The IG flag threshold was raised to 5% after detailed chart review; this was also the instrument vendor's default recommendation for the newer hematology analyzers. Analysis also supported switching to PLT-F for all platelet analysis. Autoverification rates increased to 93.5% (out of 91,692 orders) for CBC as a panel and 89.8% (out of 11,982 orders) for individual components after changes in rules and laboratory practice.

Conclusions:

Detailed analysis of autoverification of hematology testing at an academic medical center clinical laboratory that had been using a set of autoverification rules for over a decade revealed opportunities to optimize the parameters. The data analysis was challenging and time-consuming, highlighting opportunities for improvement in software tools that allow for more rapid and routine evaluation of autoverification parameters.

Chronometric vs. Structural Hypercoagulability

Prolonged tourniquet stasis induced by venepuncture can lead to the release of the plasma of cell lysis products, as well as tissue factor (TF), impairing the quality of coagulation test results. The accidental presence of TF in vitro can trigger the coagulation mechanism, generating a false decrease in prothrombin time (PT). Background and Objectives: Identification of short PT tests below the normal reference value that could suggest a situation of hypercoagulability. The study aimed to compare the results of the shortened PT tests at their first determination with the eventual correction following duplication of the analysis from the same sample. Materials and methods: Identification of the shortened PT tests has been carried out for a period of 4 months, upon 544 coagulation samples referred to the Hematology department of Sf. Spiridon County Clinical Emergency Hospital from Iasi, Romania. Results: Out of the 544 samples of which the results indicated a state of hypercoagulability, by repeating the determination from the same sample, for 200 (36.76%) PT tests (p = 0.001) the value was corrected, falling within the normal reference range. For 344 (63.24%) tests, the results suggested a situation of hypercoagulability. Conclusions: In order to guarantee the highest quality of the laboratory services, a proper interpretation and report of the patients' results must be congruent and harmoniously associated to the actual clinical condition of the patient. Duplication of the PT determination from the same sample would exclude situations of false hypercoagulability and would provide significant improvement for the patient's safety.

Using machine learning to develop an autoverification system in a clinical biochemistry laboratory

OBJECTIVES

Autoverification systems have greatly improved laboratory efficiency. However, the long-developed rule-based autoverfication models have limitations. The machine learning (ML) algorithm possesses unique advantages in the evaluation of large datasets. We investigated the utility of ML algorithms for developing an artificial intelligence (AI) autoverification system to support laboratory testing. The accuracy and efficiency of the algorithm model were also validated.

METHODS

Testing data, including 52 testing items with demographic information, were extracted from the laboratory information system and Roche Cobas^® IT 3000 from June 1, 2018 to August 30, 2019. Two rounds of modeling were conducted to train different ML algorithms and test their abilities to distinguish invalid reports. Algorithms with the top three best performances were selected to form the finalized ensemble model. Double-blind testing between experienced laboratory personnel and the AI autoverification system was conducted, and the passing rate and false-negative rate (FNR) were documented. The working efficiency and workload reduction were also analyzed.

RESULTS

The final AI system showed a 89.60% passing rate and 0.95 per mille FNR, in double-blind testing. The AI system lowered the number of invalid reports by approximately 80% compared to those evaluated by a rule-based engine, and therefore enhanced the working efficiency and reduced the workload in the biochemistry laboratory.

CONCLUSIONS

We confirmed the feasibility of the ML algorithm for autoverification with high accuracy and efficiency.

General position of Croatian medical biochemistry laboratories on autovalidation: survey of the Working Group for Post-analytics of the Croatian Society of Medical Biochemistry and Laboratory Medicine

Introduction

Autovalidation (AV) is an algorithm based on predefined rules designed, among others, to automate and standardize the postanalytical phase of laboratory work. The aim of this study was to examine the overall opinion of Croatian medical biochemistry laboratories regarding various aspects of AV.

Material and methods

This retrospective study is an analysis of the responses of a survey about AV comprised of 18 questions, as part of Module 10 (“Postanalytical phase of laboratory testing”) of national External Quality Assessment program, administered by the Croatian Centre for Quality Assessment in Laboratory Medicine. Results were reported as percentages of total number of participants in survey or as proportions of observed data if the overall number of data was <100.

Results

121 laboratories responded to the survey, of which 76% do not use AV, while 11% of laboratories use AV in routine laboratory work. 16/29 laboratories implemented semi-automated AV for general biochemistry (7/29), haematology (5/29), and coagulation (4/29) tests. Analytical measurement ranges, critical values, flags from analysers, interference indices and delta check were the most commonly used rules in the algorithm. 12/29 laboratories performed validation of AV with less than 500 samples (8/29). 7/13 laboratories report the percentage of AV being 20-50%, while 10/13 answered that introduction of AV significantly reduced turnaround time (TAT) (for 20 - 25%), especially for biochemistry tests.

Conclusions

Despite of its numerous benefits (i.e. shorter TAT, less manual validation, standardization of the postanalytical phase), only a small number of Croatian laboratories use AV.