"AI models that enhance interpretation and automate routine and time-consuming tasks can help radiologists manage challenges including increased demand for imaging tests, staffing shortages and physician burnout," said Parisa Kaviani, MD, a radiology research fellow at Massachusetts General Hospital. "But reliable clinical use of those tools also requires that radiologists have confidence in the relevance and robustness of an algorithm's training data, a model's diagnostic performance, and the compatibility of its output with reporting standards and preferences."
Kaviani
The findings of a new IRB-approved and HIPAA-compliant study indicating near-perfect agreement between AI-generated auto-impressions (AAI) and radiologists-dictated impressions (RDI) in thoracic imaging reports may ease those concerns and increase radiologists' comfort when adopting time-saving AI tools.
For the study, a team of physicians at Mass General Brigham tested an AI-based auto-impression model from a training dataset of 9.8 million radiology reports for multiple imaging modalities, radiology subspecialities, and sites developed by Nuance Communications. The team then used the model to process 12,000 de-identified reports of chest radiographs, CTs and MRIs from five thoracic radiologists.
Two other thoracic radiologists conducted a comprehensive comparison of RDI and AAI including trajectory, incidental and significant findings, and any differences in size, location, or laterality. They also evaluated how well AAI output matched RDI reporting style and word count. Data were analyzed with Pearson correlation and paired t-tests.
Overall, there was excellent correlation (r= 0.96) between RDI and AAI for classifying negative, stable, increased, and decreased findings. There were no significant differences between the number of incidental (r= 0.82) and significant (r= 0.9) findings. AAI had perfect (100%) matching of the location and side of findings, with 2% of RDI including side and location mismatches. Stylistic agreement was either perfect (87.8%) or acceptable (11.7%) with just 0.5% rated as unacceptable. Report word counts were identical.
Agarwal
Just 4% of AAI output required major changes. Those occurred in reports with multiple impression items and those where differential diagnosis or disease trajectories were not described in radiologists' findings.
While the study examined thoracic imaging reports, its overall findings of near-perfect similarity between AAI and RDI could increase radiologists' confidence in using AI models in other areas.
"The study may also help radiology practices establish performance metrics for validating AI developers' models prior to purchase and deployment," said Sheela Agarwal, MD, MBA, chief medical information officer at Nuance Communications. "The confidence gained in turn could increase adoption of AI tools to improve reporting efficiency, reduce the burdens of staff shortages and increase patient access to imaging even as volumes grow."
Visit Nuance at Booth 3300 during RSNA 2022 to see product demonstrations and get more information about Nuance diagnostic imaging solutions.
