Using a Web-based Application to Automatically Identify Discrepancies in Preliminary Interpretations Provided by Radiology Residents during Independent Call
 
Authors:
Jason N. Itri, MD, PhD, Hospital of the University of Pennsylvania; William W. Boonn, MD; Regina O. Redfern; Mary H. Scanlon, MD; Tessa Cook, MD, PhD; Woojin Kim, MD
 
Hypothesis:
Our hypothesis is that a web-based application and data-mining tool can be used to query the radiology information database (RIS) for interpretations provided by radiology residents during independent call, and can accurately identify minor and major discrepancies based on a natural language processing algorithm.
 
Introduction:
Improving quality and safety in radiology is becoming increasingly important given the dramatic conclusions of the Institute of Medicine’s report “To Err is Human,” which states that an estimated 100,000 lives per year are lost due to medical errors. It is common practice in academic hospitals for on-call radiology residents to provide preliminary interpretations during off-hours, which can potentially lead to medical errors, given that residents are less experienced and referring physicians rely on radiology residents for interpretations. Missed and misinterpreted findings can have profound ethical, legal, and psychological consequences. As a result, there are numerous prospective and retrospective studies evaluating discrepancy rates between radiology residents and attending physicians, as well as outcomes. Most of these studies define major discrepancies as those that have the potential to alter patient management in terms of diagnosis, treatment, disposition, or outcome. In contrast, minor discrepancies are not expected to impact patient management.

Major discrepancy rates between radiology residents and attending physicians range from 0.3 to 13% and appear to be dependent on modality and level of resident training, although most studies report a major discrepancy rate of 0.5 to 2%. A quality assurance program comparing initial interpretations by board-certified and board-eligible radiologists to subsequent interpretation by a subspecialty-trained abdominal imaging radiologist found a similar major discrepancy rate of 2.3% in abdominal and pelvic CT examinations performed in the emergency department. This suggests that radiology residents overall perform similarly to board-certified and board-eligible radiologists when interpreting on-call studies from the emergency department. No significant difference in patient outcomes as a result of radiology residents providing preliminary interpretations on-call has been reported.

Valid and reliable assessment of radiology resident discrepancies on-call is an important part of an academic radiology training program, both in terms of ensuring high-quality patient care and providing a rich and comprehensive educational experience. Manually grading and tracking of minor and major discrepancies, however, can be an extremely time-consuming process, particularly if this functionality is not integrated into the PACS or RIS. In the studies mentioned previously, cases with minor and major discrepancies were either recorded by hand or manually entered into an electronic database at the time of interpretation, which can be difficult or impossible in a busy department with limited resources.

The purpose of this study was to develop a web-based application capable of automatically querying the RIS database for preliminary interpretations provided by radiology residents during independent call and to accurately identify discrepancies based on an algorithm that uses the presence of specific words or phrases in the faculty-modified report and overall change in text volume.

 
Methods:
At our institution, we developed a database application called Minerva that accesses the RIS database, identifies all preliminary interpretations provided by residents during independent call, and allows single-click grading of minor and major discrepancies. Minerva was used over a one-year period (July 2008 through June 2009) to review faculty-modified exams performed in the emergency department. The resident program director reviewed all modified reports and classified them as “in agreement,” minor discrepancy, or major discrepancy, according to the guidelines described previously.

The Minerva database of reports categorized as in agreement, minor discrepancy, and major discrepancies was reviewed to identify specific words, phrases, and changes in overall text volume that could be used to reliably identify reports with minor and major discrepancies. Twenty-one specific words and phrases related to review of the preliminary report and notification of the referring clinical or service after faculty review were found to be specific to discrepant cases and when used to generate a weighting system based on the frequency of these words and phrases in each category of report. Additionally, the overall change in the text volume of the faculty-modified report was used as an independent value in discriminating reports with minor and major discrepancies.

The Minerva database was subsequently exported into an SQL database containing 1,253 records and containing both preliminary and faculty-modified reports. The results showed 201 reports were previously categorized as “in agreement,” 684 reports as minor discrepancies, and 368 reports as major discrepancies. A web-based application was used to query the SQL database and employ a formula incorporating weighted word/phrase values and change in text values to generate an overall score for each of the 1253 records.

 
Results:
When using a specific threshold, the web-based application positively identified 964 of the 1253 records. Of those 964 records, 333 were previously categorized as major discrepancies, constituting 90.5% of the total number of major discrepancies (n=368). 534 records identified were previously categorized as minor discrepancies, representing 78.1% of the total number of minor discrepancies (n=684). The remaining 97 reports were previously categorized as in agreement, representing 10.1% of the overall number of records positively identified by the web-based application.
 
Discussion:
The ability to query the RIS database and automatically identify minor and major discrepancies in interpretations provided by on-call radiology residents is a valuable and time-saving tool that can be used to assess resident performance, establish benchmarks, and track trends in missed cases to identify programmatic and training deficiencies. In the first version of our Minerva application, each report was checked manually to assess for minor and discrepancy. For this project, we used an algorithm based on those identified and validated minor and major discrepancies and were able to accurately identify greater than 90% of the major discrepancies over a year period. Additionally, we were able to identify 78% of the minor discrepancies with a low false positive rate of 10%. A majority of the false positive cases tested positive due to substantial increases in report text volume during faculty modification. Since this web-based application was designed to query an SQL database, it can be easily configured to work with any RIS database that uses a similar language to store report text.
 
Conclusion:
We have developed a web-based application capable of querying the RIS database for interpretations provided by radiology residents during independent call and accurately identifying minor and major discrepancies based on an algorithm that uses the presence of specific words or phrases in the faculty-modified report and change in text volume.
 
References:
1. Tieng N, Grinberg D, Li SF. Discrepancies in interpretation of ED body computed tomographic scans by radiology residents. Am J Emerg Med. 2007;25(1):45-8.

2. Meyer R, et al. Discrepancy rates of on-call radiology residents’ interpretations of CT angiography studies of the neck and circle of Willis. AJR Am J Roentgenol. 2009;193(2):527-32.

3. Wechsler R, et al. Effects of training and experience in interpretation of emergency body CT scans. Radiology. 1996;199(3):717-20.

4. Yoon LS, et al. Evaluation of an emergency radiology quality assurance program at a level I trauma center: Abdominal and pelvic CT studies. Radiology. 2002;224(1):42-6.

5. Wysoki MG, et al. Head trauma: CT scan interpretation by radiology residents versus staff radiologists. Radiology. 1998;208(1): 125-8.

6. Carney E, et al. Preliminary interpretations of after-hours CT and sonography by radiology residents versus final interpretations by body imaging radiologists at a level 1 trauma center. AJR Am J Roentgenol. 2003;181(2):367-73.

7. Ruchman RB, et al. Preliminary radiology resident interpretations versus final attending radiologist interpretations and the impact on patient care in a community hospital. AJR Am J Roentgenol. 2007;189(3):523-6.

8. Pomerantz SM, et al. Quality assurance for abdominal CT: A rapid, computer-assisted technique. AJR Am J Roentgenol. 1996;167(5):1141-5.

9. Erly WK, et al. Radiology resident evaluation of head CT scan orders in the emergency department. AJNR Am J Neuroradiol. 2002;23(1):103-7.

10. Cooper VF, et al. Radiology resident interpretations of on-call imaging studies: the incidence of major discrepancies. Acad Radiol. 2008;15(9): 1198-204.

11. Walls J, et al. The DePICTORS Study: discrepancies in preliminary interpretation of CT scans between on-call residents and staff. Emerg Radiol. 2009;16(4): 303-8.