Validation of quantitative assessment of florbetaben PET scans as an adjunct to the visual assessment across 15 software methods

Alzheimer’s disease (AD) is the leading cause of dementia and constitutes 60–80% of dementia cases over the age of 65 years. There is a protracted preclinical (asymptomatic) period during which abnormal traits manifest in the central nervous system detectable via biomarkers of the disease. It is characterized by the persistent formation of amyloid plaques and the subsequent development of amyloid-dependent tau pathology, neuroinflammation, and synaptic dysfunction, followed by clinical symptoms [1]. As such, the protracted AD continuum is accompanied by increasing severity of symptoms and is ultimately fatal. Detecting and quantifying such biomarkers are critical for early diagnosis of AD, for the stratification of patients for targeted disease-modifying drugs, and to monitor the effects of potential treatments by providing information about the level of relevant neuropathology [2‐5].

Biomarkers and methods for AD pathology detection have been established in recent years, which induced a shift towards a biomarker-assisted diagnosis. Guidelines emphasize the fundamental role of amyloid in the AD diagnostic process. Amyloid positron emission tomography (PET) with florbetaben (FBB) is an established tool for detecting Aβ deposition in vivo. FBB underwent a global multicenter development program and was approved by the European Medicines Agency (EMA) and the US Food and Drug Administration (FDA) in 2014, and other national agencies have subsequently approved its use [6, 7]. FBB has been robustly validated against histopathological confirmation of neuritic Aβ plaque density in the brain as the standard of truth (SoT) [8]. Visual inspection of the FBB PET scans is the approved and validated method for image interpretation and has proven to be very reliable in clinical practice [9].

Although visual assessment (VA) is an appropriate method for the vast majority of scans assessed in clinical practice, it has been proposed that certain situations would benefit from additional quantitative information gleaned from the amyloid images [10, 11]. In a heterogeneous clinical population, for example, VA can be challenging, especially for less-experienced readers: anatomical abnormalities, such as cortical thinning caused by atrophy and ventricular enlargement, may hamper VA; the presence of other neurodegenerative disorders can also confound the assessment on a purely visual basis; and the dichotomous readout of VA may lack the required sensitivity to assess longitudinal changes of amyloid load [12, 13]. A substantial part of the clinical population now includes patients with early-stage disease, which may only show isolated regional uptake and emerging amyloid deposits [14, 15]. Additionally, a small fraction of PET images is only assessed with low confidence, e.g., when amyloid levels are intermediate [16, 17]. In such situations, the continuous measurement of amyloid burden with quantification can provide additional information to the binary VA and increase confidence in such equivocal situations.

Indeed, quantification of PET images is commonly performed in research studies [18] and therapeutic clinical trials [3, 19‐24]. The recent FDA accelerated approval of lecanemab, an anti-amyloid antibody, was based on the observed reduction of Aβ plaques as quantified and monitored by amyloid PET [25]. Detection of longitudinal changes using quantification methods is established in the research and development setting [14] with many PET software packages capable of calculating amyloid burden both on a composite and a regional level [26, 27].

In this context, this retrospective data analysis was conducted to evaluate FBB PET quantification as an adjunct to VA. This study was aimed at providing scientific evidence of the robustness and additional value of FBB PET quantification. FBB PET scans from previous clinical trials were quantitatively assessed with several analytical methods. The diagnostic performance (i.e., sensitivity and specificity) of quantification against the histopathological confirmation of Aβ load was estimated and compared to the effectiveness of the approved VA method. Additionally, the concordance between visual and quantitative evaluation of FBB PET scans was assessed. The reliability and comparability of the different analytical methods were further tested.

More than 11,000 PET scan quantifications from 589 subjects were analyzed in this retrospective analysis. All scans underwent quality control to ensure correct positioning of regions of interest (see supplementary material); 97.8% (range 94.95–100%) of the cases of the full sample met the stringent quality control criteria and were included in the presented analyses.

The results of this comprehensive analysis show that all investigated software quantification methods generate homogeneous and robust data and that quantification methods can complement visual assessment of FBB PET images. Adjunct use of quantification tools could be beneficial for newly trained or inexperienced operators, in instances when images are assessed with relatively low confidence based on visual assessment alone, to detect early amyloid deposition, or when amyloid levels of patients are close to “pathology” thresholds.

The dichotomous quantitative assessment of FBB PET had a very high agreement of  > 92% to the majority VA. This result is consistent with previously published FBB data where percent agreement was reported between 88 and 97% [9, 10, 30, 41‐47]. The average concordance of quantification and VA was higher (97.4%) in the consensus subset where images were only included when all five independent readers provided the same visual assessment. Very high agreement rates between quantification and VA have previously been shown for all amyloid PET tracers [5]. It is worth noting that typically quantitative methods tend to have higher concordance rates with expert readers than with non-expert readers [48].

While some details of individual quantification methods may differ, the software evaluation revealed homogenous performances in the current study. All software packages achieved excellent diagnostic efficacy when compared to histopathology and high concordance with visual assessments. Fleiss’ kappa between analytical methods was almost perfect and superior to Fleiss’ kappa of VA (0.9 and 0.79, respectively). This substantial inter-software agreement confirms good reproducibility of the results independent of the quantification method. The pairwise kappa agreement analysis led to similarly high agreement rates. Furthermore, all the quantitative metrics provided by the different analytical tools showed a tight linear agreement with an average correlation coefficient between pairs of analytical methods of 0.95 ± 0.03. This substantial correlation between different methods substantiates the generalisability of the obtained results, and this was confirmed by an additional analysis with a publicly available dataset (GAAIN) in which quantification results from the evaluated methods correlated closely with the standard centiloid method published previously [29]. Intra-software reproducibility was excellent for all methods and correlation coefficients for the individual method re-analysis ranged between 0.98 and 1.00. Overall, the relatively heterogenous analysis approaches of the different methods yielded very homogenous and robust results.

While all approved [¹⁸F]-labeled amyloid PET tracers have been validated based on their performance in late-stage disease patients against histopathology [8, 49, 50], the performance of quantitative methods in earlier stages of disease is important for the clinical routine situation. This was tested in cohort #3 (MCI), and the results seem particularly noteworthy, as they allow direct assessment of PET quantification in patients with early disease. Positive Aβ MCI patients in this cohort were associated with significant increase in risk of clinical progression, while negative Aβ patients, defined by the pathology-derived cut-off specific to each pipeline, did not progress to AD. The results of the different quantification methods and visual assessment were comparable demonstrating very high reliability and reproducibility and in concordance to the previously reported values for FBB [7, 28, 51, 52]. This confirms that quantification and visual assessment of FBB PET scans in early AD stages provide important information to predict progression to AD in MCI subjects over a 4-year observation period. This data is in line with results of the other amyloid tracers [53‐56].

For all 15 analytical pipelines, subjects with established Aβ deposition at baseline had annual rates of Aβ accumulation statistically different from zero. Thus, quantification of amyloid load enables the evaluation of increased amyloid accumulation over time in a sensitive, objective manner; this can be particularly useful for low amyloid level subjects at an early disease stage and may help to identify groups of subjects most likely to benefit from early disease detection and possible therapeutic intervention. Indeed, selection by amyloid PET imaging has been a requirement to participate in therapeutic trials for many years [3, 4, 19‐24].

Those software packages that provided centiloid values reported similar cut-offs and interpretation of the results. Centiloid values above 35 CL indicate established Aβ pathology corresponding to moderate and frequent neuritic plaques by neuropathology. Subjects above this cut-off have a high probability of moderate or frequent plaques. Likewise, the agreement between VA and quantitative methods was extremely high for subjects with substantial amyloid levels. A lower cut-off of around 20 CL derived from elderly Aβ-negative, cognitively normal subjects provided a high specificity to rule out amyloid pathology. As for subjects with substantial amyloid deposition, agreement between VA and quantitative methods was very high for low amyloid levels. The level of agreement between VA and quantitative methods decreased for cases with intermediate amyloid accumulation or those near the pathology-derived cut-offs. This observation is consistent with the findings of Zeltzer et al., who reported a higher frequency of visual-quantitative discrepancies in scans near the amyloid positivity threshold [48]. Furthermore, subjects with CL values indicating intermediate amyloid deposition had a higher rate of not reaching full concordance between all 5 readers (see supplemental Fig. 1). These findings indicate that subjects with emerging or intermediate amyloid burden may need more careful inspection, benefiting from additional quantification derived information. Figure 5 is an illustrative example of the use of quantitative information to supplement visual assessment of FBB PET scans using centiloids.

As illustrated in Fig. 5, focal uptake is one factor that should be considered when interpreting global quantitative measures. Multiple studies and cohort types have demonstrated the robustness of centiloid quantification as a reliable measure [5, 52]. In the clinical implementation context, it is crucial to emphasize that the interpretation of quantitation should always be done in conjunction with a visual read. This ensures data quality, allows for understanding of possible causes of discordance, and helps prevent false-negative and false-positive classifications based solely on quantification or visual assessment. Figure 6 showcases examples of challenging cases characterized by emerging or focal amyloid deposition, which may result in discordance between visual readers and centiloid quantification. Most of the readers assessed these scans as positive, which were negative based on global binary quantitative assessments even though three of the cases present CL values pointing towards emerging Aβ pathology. In all these cases, regional information, e.g., as provided by z-score information, may provide additional incremental value and thus assist VA. As illustrated with these difficult and discordant patients in this work, focal uptake in regions less represented in the CL mask can result in lower quantitative values. In all these cases, regional information, e.g., as provided by z-score information, would provide an incremental value and thus assist correct classification of scans. This is in line with previous work [15, 26, 52, 57, 58].

The obtained centiloids of the current study agree with several recent reports across different tracers converging to the use of two cut-offs for amyloid PET abnormality: an early cut-off around CL = 11–20 where pathology may be emerging and a second around CL = 29–36 where amyloid burden levels correspond to moderate and frequent neuritic plaques. Bullich et al. developed a lower FBB PET cut-off of 13.5 CL when derived from young healthy controls indicating emerging Aβ pathology and derived a higher cut-off of 35.7 CL where amyloid burden levels correspond to established neuropathology findings [14]. The publication showed that these two cut-offs define a subset of subjects (13.5–35.7 CL) characterized by pre-AD dementia levels of amyloid burden that precede other biomarkers, such as tau deposition or clinical symptoms and accelerated amyloid accumulation. Other early cut-offs of 11, 14, or 17 CL have been reported for the FACEHBI, ALFA + , and AMYPAD PNHS (Prognostic and Natural History Study) studies using Gaussian mixture models [59]. Similarly, Salvadó et al. identified two cut-offs based on a direct comparison with established CSF Aβ42 thresholds: CL = 12 to rule out amyloid pathology and CL = 29 to denote established pathology [60]. Mormino et al. also showed the biological relevance of slight ¹¹C-PIB elevations in elderly normal control subjects and provided an estimate for the cut-offs defining the “gray zone” using distribution volume ratios [17]. Using histopathological confirmation, Doré et al. and La Joie et al. reported gray zones from 12.2 to 24.4 and 19 to 28 CLs, respectively [61, 62]. Although this study achieved tightly correlated results between pipelines, especially between the different centiloid methods, it should be highlighted that even for the different centiloid approaches, a range of different cut-offs was observed, which is in line with previous reports that investigated the sensitivity of centiloid quantification to pipeline design [63, 64]. The variability in the cut-offs is not attributable to random test-retest errors, but to differences in pipeline-specific factors such as spatial processing, ROI definition, and SUVR to CL calibration bias. One centiloid pipeline used in the manuscript, the standard centiloid, adopts the approach developed by Klunk et al. (2015) and employs SPM8 for PET-MRI coregistration and normalization on the standard template, followed by the use of standard centiloid ROIs [31]. However, the other centiloid pipelines may employ different spatial processing algorithms and ROI definitions, leading to increased variability beyond the expected test–retest variability and resulting in different cut-offs for each pipeline. Additionally, the number of subjects excluded by quality control measures was low but slightly varied between pipelines, possibly leading to further increases in variability.

Another limitation of the present study is that only indirect evidence on the adjunct value of quantification to visual assessment is provided. Such additional value can only be demonstrated in a prospective study. Furthermore, we acknowledge that an end-of-life cohort is less representative for early disease stages with lower amyloid burden. The derived sensitivity, specificity, and accuracy both for quantitative methods and readers were based on histopathology derived from such an end-of-life cohort, and these numbers should be considered in this context. However, similar or even better quantification performance is expected in an earlier study population, as fewer structural brain abnormalities are typically observed that could interfere with the assessments.

This study focused on simple binary reads for VA, which is the routine clinical method. However, recent research has shown the potential significance of amyloid PET regional quantification in staging AD [26, 27, 65], determining the risk of subsequent cognitive decline [27, 53], for optimal patient selection for anti-amyloid intervention trials [65, 66], and for reducing the sample size in anti-amyloid intervention trials [67, 68]. Pascoal et al. also showed that the topographical pattern of the PET signals in individuals with MCI who progress to dementia is “traditionally AD-like,” while that of non-converters includes more temporal and occipital regions instead [69]. The use of CL and composite SUVR is limited in determining the topographical distribution of Aβ load. Regional information may provide additional information to supplement visual assessment, but it has not been assessed in this analysis and is not widely available in all the quantitative tools. Such topographical information of Aβ load may enable earlier identification of subjects in the early-stage AD pathological continuum and may overcome simple dichotomous measures [15].

Finally, this study has also assessed global measurement of Aβ load such as SUVR, centiloids, amyloid load, or amyloid index. Some amyloid PET analysis software provides additional tools, such as z-scores obtained when comparing FBB PET signal from a scan with a normal database, which may assist with the clinical diagnosis. Such tools may offer additional information to supplement VA but were not evaluated in this study. Figure 6 presents several cases where regional z-score information may provide incremental value and thus assist VA.

Visual binary reads of amyloid load provide the essential information for clinical routine diagnosis of AD but do not consider the wealth of information that brain PET scans provide. This study demonstrated that quantitative methods, using both CE marked software and other widely available processing tools, perform very homogenously and robustly, providing comparable results to visual assessments of FBB PET scans. Adjunct use of quantification could be beneficial in certain situations, e.g., for newly trained or inexperienced readers, in instances when images are visually assessed with relatively low confidence, for the early detection of amyloid load or when amyloid levels are close to “pathology” thresholds, for which assessment based on VA can be difficult. Physicians should retain the careful visual inspection of the images, but quantification methods may provide additional insights in cases of doubt or for research purposes.