How to evaluate perfusion imaging in post-treatment glioma: a comparison of three different analysis methods

Ethical approval was waived by our institution’s ethical review board due to the retrospective nature of this study. We evaluated patients with suspected tumor recurrence between January 2021 and May 2022. Inclusion criteria comprised: I) 18 years of age or older with histopathologically proven glioma, II) treated with surgical resection, chemotherapy and radiation therapy, III) enlarging or new contrast-enhancing mass on follow-up MRI, IV) available pre-contrast T₁-weighted images, post-contrast T₁-weighted images and DSC PW-MRI perfusion weighted images and V) histopathological examination after lesion biopsy or surgical resection or clinical and radiological follow-up to determine the ground truth of the new enhancing lesion (TRA vs. TP). Exclusion criteria included: I) non-contrast enhancing tumor on MRI, II) distortion of MR signal due to blood products or other artefacts, III) histopathological entities other than glioma and IV) patients that opted-out of participating to research. As the inclusion criteria require a new-enhancing mass to be visible on follow-up MRI the duration until the follow-up scan used in the analyses can differ.

To determine if the final state of a lesion was TRA or TP, two methods were used in this study. If available, the gold-standard of histopathological confirmation was chosen as the primary means of determining the state of a lesion. If pathological material was unavailable, the often-used silver standard confirmation was used, consisting of clinical and radiological follow-up over a period of at least 3 months. The approach of clinical and radiological evaluation of the lesions over time complied with the RANO criteria [2].

All MR imaging was performed on a 1.5 Tesla scanner (Magnetom Avanto Fit, Siemens Healthineers, Erlangen). The protocol included a 3D T₁-weighted MPRAGE sequence (repetition time (TR) 2100 ms; echo time (TE) 2.42 ms; Inversion time (Ti) 900 ms; isotropic voxel size 1 mm; acquisition time 5 min) before intravenous injection of gadoteric acid (Dotarem, Guerbet, Villepinte) as contrast agent. Then, a preload bolus of gadoteric acid contrast agent was administered by continuous flow of contrast (Dotarem 0.1 ml/kg bodyweight at a flow rate of 5 ml/sec). After this an axial multi-slice 2D T₂-weighted sequence (TR 5310 ms; TE 85 ms; resolution 1 × 1x5 mm; acquisition time 2:30 min) and a 2D Fluid-Attenuated Inversion Recovery (FLAIR) sequence (2D, TR 9000 ms; TE 87 ms; inversion time 2500 ms; resolution 1 × 1x5 mm; acquisition time 3:30 min) was acquired. Then a bolus of contrast (20 ml Dotarem at 5 ml/sec) was injected, followed by a T₂*-weighted sequence (TR 1350 ms; TE 43 ms; voxel size 1.8 × 1.8x5 mm;acquisition time 2 min) and a T₁-weighted 3D fast turbo spin echo sequence (TR 600 ms; TE 7.1 ms; isotropic voxel size 1 mm; acquisition time 3:26 min) The susceptibility induced signal loss on T₂*-weighted sequences, resulting from the passage of the contrast agent, was used to acquire the DSC PW-MRI data.

Three radiologists or radiology residents (between 5 and 23 years of experience in neuroimaging) were tasked with placing a 70 mm² circular ROI on the perceived ‘hot spot’, as seen in Fig. 1. The readers were also asked to place a reference ROI of the same size in the contralateral centrum semiovale [16]. This reference placement has been shown to result in the most robust results when analysing perfusion metrics between different observers [17]. The mean rCBV value retrieved from the ROI placed on the ‘hot spot’ of the lesions was recorded as rCBV_hotspot, while the reference was rCBV_ref. The rCBV ratio was then calculated by dividing the rCBV_hotspot by the rCBV_ref. This was done to take the variation in base hemodynamics between patients into account, making the produced rCBV ratio a more standardized measure of perfusion.

A workstation equipped with OsiriX MD (Version 12.0; http://​www.​osirix-viewer.​com) and a commercially available plug-in (IB Neuro; Imaging Biometrics, Elm Grove, Wisconsin), which uses a leakage-correction algorithm to process perfusion data and calculate perfusion maps [15, 18, 19] was used for this analysis. For semiautomated image analysis, we used IB Rad Tech (Imaging Biometrics, Elm Grove, Wisconsin), which is a workflow engine that plots rCBV maps in semi-automatically defined volumes of interest [18, 20, 21]. A standardisation of the cerebral blood volume map was automatically performed by IB Rad Tech using a standardisation protocol developed by Bedekar et al. [22]. To visualise the T1-enhancing lesion more clearly IB Rad Tech first subtracted the intensities of the standardized pre-contrast T1 from the standardized post-contrast T1, creating a ΔT1-weighted sequence. This was followed by the semi-automatic selection of defined volumes of interest, which concerns the manual drawing of an ROI around the ΔT1-weighted area of interest, after which the protocol automatically selected the voxels that showed contrast enhancement in the drawn ROI, thus creating a complete selection of the volume of interest as seen in Fig. 1. Mean and maximum rCBV values and volumetric parameters of the contrast enhancing VOIs were automatically generated by the software package for each lesion. This semi-automatic analysis was done by a single radiology resident (D.H.) with profound experience using the software. This approach was chosen because the standardization protocol of the software package is known to produce highly reproducible rCBV values as it does not rely on manual reference placement and compensates for variability in acquisition differences [22‐24]. Secondly, the semi-automatic VOI selection based on contrast-enhancing pixels in the ΔT1-weighted sequence left little room for inter-operator variability, as it is only susceptible to other contrast enhancing artifacts such as blood vessels, which trained radiologists should be able to identify.

Three radiologists or radiology residents with between 5 and 23 years of experience in (experimental) neuroimaging assessed the DSC PW-MRI data visually or by drawing an ROI and recorded their suspected outcome of the lesion, being either TP or TRA. These predictions were mainly based on the perfusion of the lesion, as it is known that features observed on conventional imaging do not contribute to accurate differentiation [3]. This analysis was based on a single imaging session, so there was no follow-up or predating imaging available during the analysis. The analysis was performed on a workstation equipped with Syngo.via VB60 (Siemens Healthineers, Erlangen, Germany). Next to the DSC PW-MRI sequences, the readers were provided with a T₁-weighted sequence pre- and post-contrast and a T₂-FLAIR sequence to aid with anatomical orientation and lesion identification. The colour scale used to analyse the PW-MRI data was spectrum 10-step, a calibrated colour scale provided in the Syngo.via program, meaning that the colours visible were chosen to fit on a set scale, meaning that the impact of the chosen colours on decision making is minimized. The predicted outcome was then compared to the definitive outcome. The definitive outcome was determined based on available clinical and radiological follow-up and/or histopathological data. The definitive outcomes were then dichotomized by one of the authors, who was still blinded to the definite outcome at this stage of the research process.

IBM SPSS Statistics for Windows (version 28.01.1, Armonk, USA) was used for most of the statical analyses of this study. To assess the variability between the different readers of the ‘hot spot’ DSC PW-MRI data the intraclass correlation coefficient (ICC) was determined for the measured rCBV ratio, rCBV_hotspot and rCBV_ref. After this, the ICC was used to assess the inter-observer variability of the rCBV ratio measured in the ‘hot spot’ analysis.

Normality of the complete VOI segmentation voxel-wise rCBV data was assessed by use of the Kolmogorov–Smirnov test. When the data was normally distributed, mean, and standard deviation values were compared between groups using the independent Student’s t-test. If the data showed a non-normal distribution, the Mann–Whitney U-test was used to compare the median and range data between patients with TRA and TP. To evaluate whether the volume of the lesion had any correlation with the rCBV values of the lesions, a Pearson Correlation analysis was carried out.

The visual assessment data was analysed using contingency tables, resulting in the sensitivity and specificity achieved per reader. Cohen’s kappa was then used as a measure for the interobserver agreement.

A Receiver Operating Characteristic (ROC) curve was created to determine a VOI-based rCBV cut-off value with a balanced sensitivity–specificity ratio, ultimately able to classify TRA and TP. The same analysis was done to determine the semi-quantitative ‘hot spot’ based rCBV ratio cut-off value, able to classify TRA and TP. In both cases the most balanced sensitivity–specificity ratio was determined by selecting the point on the ROC-curve most closely located to the top-left corner of the unit square. The corresponding rCBV cut-off value of this sensitivity–specificity ratio was then retrieved from SPSS, together with an AUROC value indicating the diagnostic accuracy. Then Delong’s test as available in the pROC package for R [25] was used to compare the AUC’s of the created ROC curves, for this analysis R was used (version 4.3.1, R Foundation for Statistical Computing, Vienna, Austria.).

In total, 50 glioma patients were included in this study. Based on the WHO classification of gliomas [26], thirty-eight patients had a medical history of glioblastoma (76%), nine patients were treated for an IDH-mutant astrocytoma grade 2–4 (16%) and three patients were previously diagnosed with an oligodendroglioma (8%). An overview of the included patients can be found in Table 1.

The ICC of the measured rCBV ratio was 0.83 (95%-CI: 0.73–0.90). While the ICC of the measured rCBV_hotspot was 0.89 (95%-CI: 0.82–0.93) and the ICC of the rCBV_ref was 0.54 (95%-CI: 0.26–0.72). The ICC of a comparison between the ‘hot spot’ rCBV ratio and the rCBV_mean from the VOI assessment, resulted in an ICC of 0.72 (95%-CI: 0.56–0.83).

The Kolmogorov–Smirnov test showed that the VOI rCBV_mean data was normally distributed (D(50) = 0.09, p = 0.20), whereas the rCBV_max and volumetric data showed a non-normal, right-skewed distribution (D(50) = 0.13, p = 0.03 and D(50) = 0.28, p < 0.001 respectively). The retrieved rCBV values and volumes per group can be seen in Table 2.

Significant differences were observed between the measured rCBV_mean and rCBV_max values in TP and TRA lesions (t(48) = 4.3, p < 0.001 and U = 168, p = 0.007 respectively). The median volumes in the TP and TRA lesions also showed significant differences between the groups (U = 175, p = 0.011). Pearson correlation analysis showed that rCBV_max value was significantly correlated with volume of the lesions (p < 0.001; r = 0.64). The rCBV_mean value, on the other hand, was not significantly correlated with volume of the lesions (p = 0.24; r = 0.17).

The radiologists or radiology residents were anonymized as reader X, Y and Z, their results can be seen in Table 3. The inter-observer agreement between reader X and Y was κ = 0.65, between reader X and Z it was κ = -0.72 and between reader Y and Z it was κ = -0.98.

The Receiver Operating Characteristic (ROC) curves can be found in Fig. 2; in both curves TP was seen as the positive state. The VOI-based analysis showed an AUROC of 0.72 (95%-CI: 0.58–0.86) and 0.82 (95%-CI: 0.70–0.94) for rCBV_max values and rCBV_mean values, respectively. Regarding rCBV_max, a cut-off value of 5.1 mL/100 g resulted in a sensitivity of 69% and specificity of 67%. A cut-off value of the rCBV_mean of 1.11 mL/100 g resulted in a balanced sensitivity and specificity value of 72% and 76%, respectively, to distinguish TP from TRA.

For the semi-quantitative ‘hot spot’ analysis an ROC curve based on measured rCBV ratio was made based on the combined analysis of the three readers. This resulted in an AUROC of 0.69 (95%-CI: 0.61–0.78). The achieved sensitivity and specificity of this analysis is 67% and 70%, based on a balanced rCBV cut-off of 1.63 mL/100 g. DeLong’s test used on the semi-automatic VOI analysis ROC curve indicated a significant difference between the AUROC of the rCBV_mean and rCBV ratio from the ‘hot spot’ analysis (Z = -2.26, p = 0.023). It also showed a significant difference between the AUROC of rCBV_mean and rCBV_max (Z = -2.655, p = 0.0079) and a non-significant difference between the AUROC of the rCBV_max and the rCBV ratio from the ‘hot spot’ analysis (Z = -0.50, p = 0.61).

In the VOI analysis, the entire T1-weighted contrast enhancing lesion was included, which limits the inter- and intra-observer variability when compared to manual placement of regions of interest. The used IB-Rad Tech software and it’s semi-automatic processing of DSC-PWI images has been described to further reduce user-related variability [22], as its automated standardisation circumvents the manual placement of references which was shown to be especially vulnerable to subjectivity. Another strength of the current study concerns the fact that for the first time, single-moment rCBV values of VOIs were directly compared with rCBV values of ‘hot spots’ and a purely visual assessment, which are commonly used in standard clinical reading of DSC PW-MRI data in the post-treatment evaluation of gliomas to distinguish TP from TRA.

An important limitation of the current study comprised its retrospective nature and the lack of an external validation cohort in which the observed threshold values can be tested. It must be emphasized that the differences between the rCBV values of the VOIs were valid on a group level; the usefulness of this technique in individual patients needs further investigation.

The single-moment analysis of the PWI data can also be seen as a limitation, as a three to six-month period of follow-up imaging and pre-operative scans are usually utilised to differentiate between TP and TRA. However, readers were informed with regard to the fact that the MR data they read were the first imaging study on which a new or growing contrast-enhancing lesion appeared. Therefore, it was not expected that this single-moment analysis impacted the outcome of the reader study.

The heterogeneity of the studied population can be seen as a limitation as they possess inherent differences in malignancy grade and therefore in behaviour and treatment options. However, all included subtypes can progress on follow-up imaging and can show new enhancing lesions in the post-treatment setting. As this study investigated the diagnostic accuracy of three approaches to analyse DSC PW-MRI data when new contrast-enhancing lesions occur, the impact of the rather heterogenous population is considered minimal.

The small number of patients per subgroup (when stratified by for example tumor type or treatment schedule) prevented a statistically sound analysis of the used subgroups and identification of potential differences between the used pathologies. As the scope of this study was aimed at elucidating differences between the used methodologies, this was deemed acceptable by the authors, however a future study with larger subgroups per pathology and treatment used should aid in the identification of potential differences.

A final limitation is the usage of a 1,5 T MRI system instead of a 3 T system. A 3 T scanner would allow for an increased signal-to-noise (SNR) ratio, increased temporal- and spatial resolution. However, DSC PW-MRI is often not limited by SNR, and the usage of 1,5 T in glioma imaging shows an almost perfect correlation with 3 T in MR modalities such as rCBV and identified lesion volume [29] so using a 1,5 T system should be sufficient in answering our study objectives.