Reliability of soft-copy versus hard-copy interpretation of emergency department radiographs: a prototype study

Effect of picture archiving and communication system on diagnosis accuracy in CT and radiography examinations in emergency departments

INTRODUCTION

Picture archiving and communication system (PACS) affects the radiologists' and physicians' performance. We aimed to evaluate the effect of implementing PACS on the emergency department (ED) physicians' accuracy compared to a radiologist's diagnosis in Iran.

METHODS

We retrospectively collected data for three six-month periods before and after the implementation of PACS on CT scan and radiography examinations. We compared ED physicians' diagnoses of CT scan and radiography images with a radiologist's interpretations for the same images. We compared 374 CT scans and 346 radiography examinations before implementing PACS (July 2015 to December 2015); 507 CT scans and 480 radiography examinations immediately after PACS (July 2016 to December 2016); and 870 CT scans and 1137 radiography examinations one year after PACS (July 2017 to December 2017).

RESULTS

We found that diagnosis accuracy of ED physicians on CT scans increased from 75.9 % before implementing PACS to 84.4 % immediately after PACS and 94.9 % one year after PACS (p-value < 0.0001). Diagnosis accuracy for radiography images increased from 63.0 % before implementing PACS to 80.2 % immediately after PACS and 93.1 % one year after PACS (p-value < 0.0001).

CONCLUSION

Implementation of PACS technology increases ED physicians' diagnosis accuracy.

Detectability of the diagonal direction in the assessment of medical images using a high-brightness liquid crystal display monitor

Liquid crystal display (LCD) monitors show a homogenous quadratic pattern in the number of pixels, size, and dimensions. However, their modulation transfer function (MTF) has a non-isotropic effect in the vertical, horizontal, and diagonal directions on the screen from the shape and pattern of arrangement of pixels. Moreover, the MTF of the human eye differs in spatial frequency response directivity among individuals. In this study, the high-brightness LCD monitor detectability was physically simulated and visually examined throughout the system, including the imaging system. Furthermore, the influence of anisotropy of the LCD monitor was evaluated. The MTF of the LCD monitor was measured by acquiring a bar pattern using a digital camera with sufficiently small pixels in the vertical, horizontal, and diagonal directions and by performing interpolation processing through waveform reproduction and frequency analysis. The detectability of the LCD monitor was verified throughout the system, including the imaging system. Radiographs of the rectangular wave chart (0.5-10 LP/mm) were obtained in the vertical, horizontal, and diagonal (45°) directions to assess the perceivable limit of the human eye (LP/mm). The spatial resolution of the LCD monitor in the diagonal direction was higher than that in the vertical or horizontal direction, which was in good agreement with the results of the profile analysis and visual evaluation.

Can paper replace laser film to communicate the results of wrist radiographs in trauma cases? A reproducibility study of the reading of wrist trauma case radiographs on a PACS workstation, laser film, and paper

The main goal of this study was to determine the reproducibility of the reading of wrist trauma case radiographs using three different media: laser film, a picture archiving and communication systems (PACS) workstation, and paper with an optimized layout. The study was conducted retrospectively in 200 consecutive patients consulting at the emergency department for wrist trauma and who underwent wrist X-ray investigation using a computed radiography system. There were 82 men and 118 women. The mean age was 48.3 years (16-95 years). Our institutional review board does not require patient approval or informed consent for retrospective review of case records. The readings were made by two independent readers who analyzed the 200 patient radiographs consecutively in one session for each type of media: paper, laser film, and on a PACS dual-screen workstation. The inter-reader agreements were substantial or almost perfect, with kappa values of 0.83 (0.76-0.90) for the PACS, 0.83 (0.76-0.90) for film, and 0.80 (0.72-0.87) for paper. The inter-technique agreement was almost perfect in all cases. There is a high interobserver agreement between PACS, laser film, and paper readings for wrist trauma cases. With a layout of one radiograph on each sheet, paper could replace laser films to communicate the results of wrist radiographs in trauma cases for outpatients.

Does PACS improve diagnostic accuracy in chest radiograph interpretations in clinical practice?

OBJECTIVES

To assess the impact of a Picture Archiving and Communication System (PACS) on the diagnostic accuracy of the interpretation of chest radiology examinations in a "real life" radiology setting.

MATERIALS AND METHODS

During a period before PACS was introduced to radiologists, when images were still interpreted on film and reported on paper, images and reports were also digitally stored in an image database. The same database was used after the PACS introduction. This provided a unique opportunity to conduct a blinded retrospective study, comparing sensitivity (the main outcome parameter) in the pre and post-PACS periods. We selected 56 digitally stored chest radiograph examinations that were originally read and reported on film, and 66 examinations that were read and reported on screen 2 years after the PACS introduction. Each examination was assigned a random number, and both reports and images were scored independently for pathological findings. The blinded retrospective score for the original reports were then compared with the score for the images (the gold standard).

RESULTS

Sensitivity was improved after the PACS introduction. When both certain and uncertain findings were included, this improvement was statistically significant. There were no other statistically significant changes.

CONCLUSION

The result is consistent with prospective studies concluding that diagnostic accuracy is at least not reduced after PACS introduction. The sensitivity may even be improved.

An approach to comparing accuracies of two FLAIR MR sequences in the detection of multiple sclerosis lesions in the brain in the absence of gold standard

RATIONALE AND OBJECTIVES

The purpose of this study was to present a new methodology to compare accuracies of two imaging fluid attenuated inversion recovery (FLAIR) magnetic resonance sequences in detection of multiple sclerosis (MS) lesions in the brain in the absence of ground truth, and to determine whether the two sequences, which differed only in echo time (TE), have the same accuracy.

MATERIALS AND METHODS

We acquired FLAIR images at TE(1) = 90 ms and TE(2) = 155 ms from 46 patients with MS (24-69 years old, mean 45.8, 15 males) and 11 healthy volunteers (23-54 years old, mean 37.1, 6 males). Seven experienced neuroradiologists segmented lesions manually on randomly presented corresponding TE(1) and TE(2) images. For every image pair, a "surrogate ground truth" for each TE was generated by applying probability thresholds, ranging from 0.3 to 0.5, to the weighted average of experts' segmentations. Jackknife alternative free-response receiver operating characteristic analysis was used to compare experts' performance on TE(1) and TE(2) images, using successively the TE(1)- and TE(2)-based ground truths.

RESULTS

Supratentorially, there were significant differences in relative accuracy between the two sequences, ranging from 8.4% to 12.1%. In addition, we found a higher ratio of false positives to true positives for the TE(2) sequence using the TE(2) ground truth, compared to the TE(1) equivalent. Infratentorially, differences in the relative accuracy did not reach statistical significance.

CONCLUSION

The presented methodology may be useful in assessing the value of new clinical imaging protocols or techniques in the context of replacing existing ones, when the absolute ground truth is not available, and in determining changes in disease progression in follow-up studies. Our results suggest that the sequence with shorter TE should be preferred because it generates relatively fewer false positives. The finding is consistent with results of previous computer simulation studies.

Influence of display quality on radiologists' performance in the detection of lung nodules on radiographs

The purpose of this study was to evaluate the influence of display quality on radiologists' performance in the detection of lung nodules. Display systems with various technical properties were considered based on their general availability in a radiology department. Their quality was assessed by physical tests. Multireader-multicase receiver operating characteristic (ROC) analysis was used to evaluate observer performance. The area under the curve (Az) was used as a metric for detectability of simulated lung nodules with diameters of 5 mm and 10 mm, and peak contrast values ranging from 0.1 (subtle) to 0.4 (evident) that were digitally superimposed on normal chest radiographs. Three experienced radiologists interpreted a batch of 60 radiographs on five different display systems; four monitors (two liquid crystal display (LCD) and two cathode ray tube (CRT) monitors) and one printed hardcopy. The physical tests showed superior performance of the two LCD monitors. ROC analysis resulted in the following Az scores: LCD-5MP Az = 0.78, hardcopy Az = 0.77, LCDc-2MP Az = 0.75, CRT-5MP Az = 0.72 and CRTc-1MP Az = 0.71. Difference in Az scores between the LCD-5MP monitor and both the CRT-5MP (p = 0.04) and CRTc-1MP (p = 0.01) monitors was significant. The primary class CRT-5MP monitor that showed reduced observer performance failed to comply with physical acceptance requirements. Luminance response was particularly observed to be insufficient. The results indicate that a quality assurance program has the potential to detect non-optimised display systems that could otherwise result in reduced observer performance.

Reliability analysis for manual measurement of coronal plane deformity in adolescent scoliosis. Are 30 x 90 cm plain films better than digitized small films?

For several years, digitized small radiographs are used to measure Cobb angle in idiopathic scoliosis. The interobserver and intraobserver Cobb angle measurement variability associated with small radiographs were compared with measurement variability associated with the long-cassette radiographs. Twenty adolescent patients with a double major idiopathic scoliosis had erect full-spine p-A radiographs and Cobb angle measurements performed by eight different observers on a 30 x 90 cm plain-film radiograph and a digitized 14 x 42 cm image. Inter-observer and intra-observer reliability using each techniques were assessed using a paired t-test, Spearman rank correlation study and intraclass correlation coefficients. The angle variability between small film and plain-film measurements was assessed using the same methods. Intra-observer and inter-observer study showed good reliability using both techniques. The comparison between small films and plain-films measurements showed very good agreement with an intraclass correlation coefficient of 95% and confidence interval between 0.962 and 0.972. In our study, Cobb angle determination was not found to vary significantly with film size. The small film image used for full-spine radiographs in our institution allows manual Cobb angle measurements to be performed. A study is currently conducted in our institution to determine if a computer-assisted measurement method significantly improves Cobb angle measurements reliability in routine practice compared with manual measurements of Cobb angles on small films.

Influence of film digitization on radiological interpretation

Our objective was to evaluate the influence of changing from analogue to digital imaging on interobserver and intraobserver image interpretation. Three radiologists interpreted 96 three image series of occipitomental radiographs of paranasal sinuses from the films and from the corresponding digitized images from the screen. Images were classified according to degree of abnormality as either normal, with mucosal thickening of less than 5 mm, with mucosal thickening of 5 mm or more, total opacity, air-fluid level or polyp or cyst of maxillary sinuses. In the present study we found that there were more differences between two radiologist's interpretations with a single method than in a single radiologist's interpretations between the methods, although radiologists interpreted fewer pathological findings from the digitized images than from the corresponding films. Our data show that the results of image interpretation are preferentially dependent on the reader rather than on the method of reading.

Flat-panel display (LCD) versus high-resolution gray-scale display (CRT) for chest radiography: an observer preference study

OBJECTIVE

Our objective was to compare cathode ray tube (CRT) display with liquid crystal display (LCD) for soft-copy viewing of chest radiographs in a clinical setting.

MATERIALS AND METHODS

We displayed 80 posteroanterior digital chest radiographs side by side on a 5-megapixel CRT display and a 3-megapixel LCD. Gradation characteristics of both monitors were adjusted to DICOM display standards. Using a 4-point scale, seven radiologists ranked overall image quality and visibility of anatomic landmarks. Data analysis included Wilcoxon's rank sum test to assess the significance of preference for the different display modes and calculation of the percentage of images ranked equally by at least five of the seven radiologists.

RESULTS

Wilcoxon's rank sum test found significant preferences (p < 0.001) for the CRT display for visualization of structures in low-attenuation areas of the thorax and for the LCD for visualization of structures in high-attenuation areas of the thorax. Overall image quality was ranked equal by at least five radiologists in 70% of cases, whereas for the remaining images a significant preference was found for the CRT display.

CONCLUSION

We conclude that, under subdued ambient lighting conditions and without use of windowing, for most images the overall quality is equal with high-resolution CRT display and LCD. In images judged preferentially, we found a significant superiority for LCD for delineating mediastinal structures and for CRT display for delineating structures in the lung.

Soft-Copy Reading of Digital Chest Radiographs

OBJECTIVE

We sought to evaluate whether soft-copy reading of simulated pulmonary chest lesions is influenced by ambient light and automatic optimization of cathode ray tube (CRT) monitor luminance.

MATERIALS AND METHODS

Four types of simulated lesions (nodules, lines, micronodules, and patchy opacities) were superimposed over an anthropomorphic chest phantom. Lesion detection with soft-copy reading was assessed using a high-contrast grayscale 2K CRT monitor under the following conditions: (1) subdued lighting (<50 lux); (2) normal lighting conditions (450 lux) without, and (3) with a sensitivity modulation to automatically adjust the CRT luminance to the increased amount of ambient light. Reading data were analyzed according to receiver operating curve. Significance of differences was tested using an analysis of variance for repeated measures.

RESULTS

Ambient room light of 450 lux did not significantly influence the detection of nodules and patchy opacities. However, bright ambient light significantly decreased detection of micronodules (0.60 vs. 0.74) and lines (0.52 vs. 0.66) relative to subdued lighting conditions. Automatic luminance adjustment could compensate the effect of ambient light for the micronodules (0.77) but not for the lines (0.53).

CONCLUSION

Bright ambient light significantly decreases detection of small low-contrast structures. This may be partially but not completely compensated by an automatic luminance adaptation.

Comparison of liquid crystal versus cathode ray tube display for the detection of simulated chest lesions

The purpose of the study was to compare the detection performance of a cathode ray tube (CRT) monitor versus a liquid crystal display (LCD) monitor for simulated subtle pulmonary lesions. Ten templates containing simulated lung lesions were superimposed on an anthropomorphic chest phantom. Posteroanterior radiographs were obtained using flat panel technology and were displayed on a CRT and an LCD monitor. Image processing and reading conditions were equivalent for both softcopy displays. Five observers assessed lesion detectability using receiver-operating characteristic (ROC) methodology. A multivariate test (Pillai trace) was used to test the significance of differences (P<0.05). The multivariate test revealed significantly different detection rates for the lesion types, but no significant difference between the two display modes. Detection performance for both monitors was higher for nodules and micro-nodules and lower for lines and patchy opacities. Analysis of lesion subgroups according to their location in lucent/obscured lung areas was also not statistically significant. Under ideal reading conditions, CRT and LCD displays perform equivalently for the detection of simulated subtle pulmonary lesions.

Information management in the emergency department

Information system planning for the ED is complex and new to emergency medicine, despite being used in other industries for many years. It has been estimated that less than 15% of EDs have comprehensive EDIS currently in place. The manner in which administration is approached in large part determines the success in obtaining appropriate institutional support for an EDIS. Active physician and nurse involvement is essential in the process if the new system is to be accepted at the user level. In the ED, large volumes of information are collected, collated,interpreted, and acted on immediately. Effective information management therefore is key to the successful operation of any ED. Although computerized information systems have tremendous potential for improving information management, such systems are often underused or implemented in such a way that they increase the workload on caregivers and staff. This is counter productive and should be avoided. In developing and implementing EDIS one should be careful not to automate poorly designed manual processes. Examples are ED tracking systems that require staff to manually relocate patients in the system. This task probably is completed only when the ED volume is low and "worked around" when the department is busy. Information from such a system is, therefore, flawed; at best useless and at worst counter productive. Alternatively, systems are available that can track patients automatically through the ED by way of infrared sensors similar to those used in baggage-tracking systems that have been in place in airports for years. In the automated (computerized) ED, we must have zero-fault-tolerant,enterprise-wide, hospital information networked systems that prevent unnecessary duplication of tasks, assist in tracking and entering data, and ultimately help analyze the information on a minute-to-minute basis. Such systems only reach their potential when they are fully integrated, including legacy systems, rather than stand alone proprietary EDIS. Further,a modular approach in which individual components are connected to a flexible computer backbone is ideal.Finally, good clinical content is key to virtually every aspect of the EDIS. Much of this content is yet to be developed and what is available still needs to be adapted to the EDIS environment. Daunting as it may be, an EDIS implementation properly accomplished results in better patient care, improved staff productivity, and a satisfying work environment (Box 3).

Impact of Ambient Light and Window Settings on the Detectability of Catheters on Soft-Copy Display of Chest Radiographs at Bedside

OBJECTIVE

The aim of this study was to evaluate how ambient light and interactive adjustment of density and contrast affect the detection of catheter fragments when interpreting bedside chest radiographs on soft-copy displays.

MATERIALS AND METHODS

A total of 131 catheter fragments were superimposed over 10 bedside chest radiographs obtained with storage phosphor technology. Images were displayed on a clinical intensive care unit viewing station (color cathode-ray tube monitor, 21 inch [53 cm], 1280 x 1024 matrix) and were independently evaluated by five radiologists. The number of catheter fragments per image varied between 12 and 14, with an approximately equal distribution in high- and low-absorption areas. Detectability of catheter fragments was assessed under subdued and bright ambient light conditions with and without interactive adjustment of window width and level.

RESULTS

Under subdued light, the detection rate of catheter fragments was significantly higher than under bright light (51.8% vs 56.6%, p < 0.05). Interactive window setting adjustment significantly increased the detection rate from 52.5% to 60.8% (p < 0.05) under subdued light and from 47.9% to 55.6% (p < 0.05) under bright light. With adjustment of window settings, the difference between the detection rates under subdued light (60.8%) and under bright light (55.6%) did not reach statistical significance.

CONCLUSION

Detection of catheters on soft-copy display is significantly decreased by bright ambient light, an effect that can be largely compensated for by means of interactive adjustment of window settings.

Accuracy of interpretation of CT scans: comparing PACS monitor displays and hard-copy images

OBJECTIVE

The purpose of this study was to determine the relative diagnostic accuracy of radiologists in the interpretation of CT scans using a computer workstation in comparison with using film.

MATERIALS AND METHODS

Four board-certified radiologists with extensive soft-copy experience interpreted 117 CT scans in four anatomic regions using films displayed on an alternator and images displayed on a four-monitor workstation. The radiologists were asked to interpret the scans in their usual fashion and were aware that both the time required to review the study and the accuracy of the reports were being assessed. The radiologists' diagnostic impressions were compared with those of a consensus panel and scored for accuracy.

RESULTS

Soft-copy interpretation using computer workstations was found to produce statistically significant improvement in combined measurements of sensitivity, specificity, and overall accuracy for chest, brain, and chest-abdominal CT scans compared with film interpretation.

CONCLUSION

PACS (picture archiving and communication system) offers radiologists the potential for improved accuracy in CT interpretation compared with traditional film-based interpretation.

Impact of pager notification on report verification times

RATIONALE AND OBJECTIVES

The purpose of this study was to assess the impact on times of verification (TOVs) by a pager notification system (PNS) that informs physicians when reports are available for signature.

MATERIALS AND METHODS

An automated PNS was implemented in the authors' department in November 2000. Monthly report verification times of each physician were collected for 3 months in the years before and after initiation of the PNS. Radiologists enrolled in the PNS and those who were not were assigned into two groups for analysis. Mean TOVs for the two sets of 3 months and for the two groups were calculated and differences recorded. Two-tailed t tests were used to assess for statistical differences between the groups.

RESULTS

Twenty-nine of 37 radiologists voluntarily enrolled in the PNS (group 1). Mean TOV was 26.75 hours (standard deviation [SD] = 17.76) for these physicians before and 14.48 hours (SD = 11.86) after the PNS was employed (P < .01). For those physicians who did not enroll in the PNS, mean TOV was 11.53 hours (SD = 5.55) before and 9.77 hours (SD = 9.86) after the PNS was employed (P = .33). Both the absolute and percentage reductions in TOVs were significantly greater for those physicians enrolled in the PNS than for those who were not (P = .035). Twenty-three of 29 (79%) physicians who used the PNS showed a reduction in their report turnaround times.

CONCLUSION

Linking the PNS with the radiology information system to notify physicians of unsigned reports was effective in reducing report verification times.