“Will I change nodule management recommendations if I change my CAD system?”—impact of volumetric deviation between different CAD systems on lesion management

The widespread routine clinical implementation of lung cancer screening programs all over the world is directly linked to the increasing importance of standardized pulmonary nodule management. Current clinical practice guidelines rely on precise and reproducible measurements of the detected pulmonary nodules. Since different definitions of the nodule diameter can be applied (e.g., mean or maximum diameter), volumetry of pulmonary nodules is deemed the most accurate predictor of lung cancer risk [1, 2] and may account for irregular shapes, air spaces, and mixed solid and subsolid compartments [3]. Furthermore, the repeatability of diameter measurements is known to be suboptimal [4]. Therefore, patient management proposed in the latest guidelines by the British Thoracic Society (BTS), the Nederlands-Leuvens Longkaker Screening Onderzoek (NELSON) group, or the American College of Radiology (ACR) is mainly based on nodule volumetry [5‐7].

In front of an increasing number of screening examinations and the associated additional workload, careful CT reading and precise measurements are challenging for radiologists in daily clinical routine. A nodule measurement variability of ± 25% has been demonstrated in several in vivo “coffee-break” studies, in which individuals were scanned twice on the same day [8, 9]. A postulated solution to tackle the increasing workload while at the same time maintaining or even improving nodule measurement accuracy is the introduction of artificial intelligence (AI)–based computer-aided diagnosis (CAD) systems into clinical routine. Further advantages of such systems comprise the automation of repetitive radiological duties and the avoidance of reader fatigue, which have been recognized as an increasing focus in the field [10]. In the context of pulmonary nodule detection, deep learning (DL)–based CAD systems have proven to be helpful as first or second reader devices [11‐13]. Since it is very likely that the majority of pulmonary nodule analyses will be performed by software and not by human readers in the near future, it seems pertinent to evaluate the performance of DL-based CAD systems in this regard. It is tempting to speculate whether novel DL-based CAD systems will have similar or lower inter-reader variability than different human individuals.

The influence of scanning parameters and reconstruction on nodule volumetry is well appreciated [14, 15]. In addition, there are studies comparing different software solutions in the chest area focusing on functional lung parameters [16]. But only few studies have compared different software packages regarding the accuracy of pulmonary nodule volumetry [17, 18].

The primary aims of this study were to evaluate the measurement accuracy of a commercially available DL-based CAD system and a standard CAD system with an in-built semi-automatic volumetry tool in comparison to the true volume of artificial pulmonary nodules and to evaluate the impact on patient management recommendations. The secondary aim was to analyze the influence of tube voltage, nodule size, and attenuation on the measurement accuracy of the CAD systems.

The results of this work showed that changing the CAD system has a potential effect on the correct pulmonary nodule classification and therefore on patient management. Hereby, the DL-based CAD had a higher initial detection rate and classified a higher proportion of solid nodules correctly compared to the standard CAD system.

Inaccurate volumetry may lead to wrong lesion management decisions, which can either delay the correct diagnosis and treatment on the one side or cause unnecessary costs on the other, especially in the context of major lung cancer screening programs. Regarding patient management, changes in lesion size and the resulting potential shift in LungRADS categorization are critical. The DL-based CAD classified 88.5% of all solid nodules correctly according to LungRADS, which was significantly higher compared to the standard CAD system (79.8%, p = 0.004). Between the systems, 14.9% of the solid nodules were classified differently (p = 0.002). These numbers have to be interpreted with caution since most of the falsely classified nodules belonged to the 8-mm-size group, which is located right at the border between LungRADS categories 3 and 4A, implying that only minor volumetric mistakes can already lead to different classifications [24]. However, the experiment indicated that patient management can be affected by changing the CAD system.

The descriptive analysis indicated no clear difference between the three tube voltage groups in the current setting. It has to be mentioned that the DL-based software did not detect 19 nodules in the images acquired with 80 kV and therefore excluded them from the analysis, which most probably explains the observed difference between 80 and 120 kV regarding the GGN.

In contrast to the current observations, a previous study indicated that the alterations of tube current and voltage have an effect on the performance of this specific DL-based CAD system in the context of pulmonary nodule detection [25]. However, the current findings are in line with various other studies, which found CAD volumetry to be robust over a range of exposure settings [14].

The comparison of the density groups revealed that the DL-based CAD system was more precise in volumetry of the GGN and less precise in measuring solid nodules than the standard CAD. Standard CAD systems classically struggle with the detection of ground-glass lesions, due to the smaller density differences between normal lung and lesion [26‐28]. The DL-based CAD system, on the other hand, was trained on both types, solid and subsolid lesions, which is reflected by the current results and is in line with their reported superiority over standard CAD systems in this regard [29].

In the analysis of the different size groups, the DL-based CAD system showed two measurement clusters regarding mainly the solid 12-mm nodules, indicating a systematic measurement error. This finding was most probably caused by the calibration dataset, which the software was trained on before the implementation; this dataset mainly contained small pulmonary nodules < 6 mm and may have led to a systematic underestimation of the larger nodule groups.

Apart from the solid 12-mm nodule group, both systems had the highest RVEs measuring the 5-mm nodules. This finding was somewhat to be expected, since only small measurement deviations can lead to high RVEs in this size group. Additionally, in this specific study, all nodules had been attached to vessels during the arrangement of the phantoms, making an accurate segmentation of the borders even more difficult. Correct segmentation of the nodule borders is crucial, as one pixel increase in this area may already alter the measured volume considerably [17]. In the case of the 5-mm GGN, the volumetric inaccuracy of the standard CAD system could even lead to the false assumption of nodule growth, since there was a volumetric error of > 25% [5, 30]

The mean RVD between the systems was 1.3 ± 18.6% for the solid nodules and – 15.2 ± 23.5% for the GGN. Bartlett and colleagues reported a mean RVD of − 0.9 ± 16.3% while assessing the interscan variability of 100 nodules measured twice with a standard CAD system [23]. For nodules < 80 mm³, they reported a mean RVD of − 0.3 ± 8.4%, which is much lower than the results for the respective group observed in the current study (10.2 ± 23.6% and – 25.6 ± 35.0% for solid nodules and GGN, respectively). A possible explanation is that Bartlett et al excluded all nodules with vascular or pleural attachment, which are more difficult to measure.

This study has several limitations. First, the protocol used is not state of the art for lung cancer screening according to the current guidelines by the European Society of Thoracic Imaging (ESTI), in particular referring to FBP as a reconstruction algorithm and the absence of overlapping image reconstruction [31]. However, the latest ACR guidelines are less strict as they only favor iterative reconstruction (IR) methods over FPB and do not recommend an overlapping reconstruction as mandatory [32]. Furthermore, it is a fact that many sites still use FBP for lung cancer screening [33], making the results of this study relevant for many institutions, especially the ones with only limited access to innovative scanner technologies [34]. In a previous phantom study, FBP and IR showed nearly identical results for pulmonary nodule volumetry using a software from the pre-AI era [14]. However, the same study setup is warranted for the AI era. Another important limiting aspect is the fact that the study was conducted on one single CT scanner with fixed settings; therefore, the results cannot be broadly transferred to other institutions or CT scanners. As already reported in various studies, the scanner settings such as reconstruction algorithm, slice thickness, etc. have a relevant impact on the performance of AI-based CAD systems [25, 35, 36]. In contrast to the current study, most of these studies were focused on pulmonary nodule detection rather than on volumetry. Another limitation is the use of artificial, perfectly spherical nodules with homogenous density and no surrounding pathology, which is of course not realistic and demands for repetition of this study in a clinical setting. However, the phantom setting of this study enabled a correlation with a perfect ground truth and an accurate evaluation of the CAD systems.

In conclusion, the DL-based system had a higher initial detection rate of pulmonary nodules and a higher proportion of them would have been classified correctly according to LungRADS compared to the standard CAD system. Nodule size and attenuation had an effect on the measurement accuracy of both systems; there was no effect of tube voltage. Our results indicate that measurement inaccuracies between CAD systems have a potential impact on patient management, which demands careful revision and, if needed, manual correction by radiologists.