Differentiating between common PSP phenotypes using structural MRI: a machine learning study

One hundred and nine PSP patients (65 probable PSP-RS and 44 probable PSP-P) were consecutively recruited at the Movement Disorder Center of Magna Graecia University, between 2012 and 2020.

The clinical diagnoses of PSP-RS and PSP-P were performed by movement disorder specialists according to international diagnostic criteria [1]. PSP patients enrolled before 2017 were diagnosed according to previous diagnostic criteria [36] and expert guidelines [37] and were retrospectively reclassified according to recent MDS diagnostic criteria for probable PSP-RS (vertical ocular dysfunction associated with early postural instability) and PSP-P (vertical ocular dysfunction associated with parkinsonism as predominant clinical features, in the absence of early postural instability) [1]. PSP-P patients with disease duration shorter than 3 years underwent clinical follow-up to rule out the appearance of early falls. Exclusion criteria were the presence of clinical features suggestive of other diseases, normal striatal uptake on 123I-FP-CIT-SPECT, and MRI abnormalities such as lacunar infarctions in the basal ganglia, diffuse subcortical vascular lesions, or imaging signs suggestive of normal pressure hydrocephalus [38]. Most PSP-P patients included in the current cohort have been reported in a recent study to validate the automated MRPI 2.0 [29], but no comparison with PSP-RS patients was done in this previous study. All patients underwent a neurological examination including the MDS—sponsored revision of the Unified Parkinson’s Disease Rating Scale part III (MDS-UPDRS-III) [39] in off-state, the Hoehn and Yahr (H–Y) rating scale [40] and the Mini Mental State Examination (MMSE) [41]. Written informed consent according to the Declaration of Helsinki for the use of their medical records for research purposes was obtained from all individuals participating in the study. All study procedures and ethical aspects were approved by the institutional review board (Magna Graecia University review board, Catanzaro, Italy).

All study participants underwent a brain MRI with a 3 T-MR750 General Electric scanner and an 8-channel head coil, with a recently described MRI protocol including a 3D T1-weighted MR image [42]. Freesurfer 7 was employed with the standard pipeline recon-all to automatically extract thickness and volume of 34 cortical regions for each hemisphere, and the volume of the subcortical regions caudate, putamen, globus pallidus, thalamus and cerebellum, divided into white and gray matter (WM, GM) [43]. All the segmentations performed by Freesurfer were visually inspected by a neuroradiologist, and images with inaccurate segmentations due to prominent movement artefacts (3 PSP-RS and 5 PSP-P patients) were excluded. The automated MRPI and MRPI 2.0 were calculated on 3D T1-weighted MR images using the previously described algorithm [29]. In 4 PSP-RS patients the algorithm failed and the MRPI and MRPI 2.0 were measured manually by an expert rater.

Difference in gender distribution was assessed with Fisher’s exact test. Normality of data was tested using Shapiro’s test. The analysis of variance (ANOVA) or Kruskal–Wallis test were employed for comparing age at examination and education level among the three groups (PSP-RS, PSP-P and control subjects). Age at disease onset, disease duration and clinical scores were compared between PSP-RS and PSP-P patients using t-test or Wilcoxon rank sum test. ANCOVA with age and education level as covariates was applied to assess differences in MMSE. ANCOVA with age and gender was used to compare cortical thickness, cortical and subcortical volumes among groups. Other covariates in the ANCOVA included: education level for cortical thickness, education level and intracranial volume (ICV) for cortical volumes, and ICV for subcortical volumes. All ANCOVA tests was repeated to assess differences between PSP-RS and PSP-P including also disease duration as covariate. All tests were two tailed, and the α level was set at p < 0.05. All p values were corrected according to Bonferroni. Statistical analysis was conducted with R language version 4.1.2.

We first assessed the diagnostic performance of the automated MRPI and MRPI 2.0 in differentiating between PSP-RS and PSP-P patients, and between patients and controls. In addition, we also tested these biomarkers in a sub-cohort of early PSP patients (38 PSP-RS and 21 PSP-P) with disease duration up to 4 years (early stage), selected from the whole cohort. Optimal cut-offs, defined as the values with the highest sum of sensitivity and specificity on the Receiver Operating Characteristic (ROC) curves, and 95% confidence intervals (CI), were calculated using pROC software package with bootstrapping (n = 2000 iterations) [44].

Subsequently, we investigated the performance of Machine Learning (ML) models based on structural MR imaging data in distinguishing between PSP-RS and PSP-P patients, and between patients and controls, both in the whole cohort and in the above-mentioned early cohort. ML models used two different tree-based algorithms (Random Forest [RF] and XGBoost) [34, 35] with all combinations of six different imaging variable groups: cortical thickness (34 regions for each hemisphere), cortical volumes (34 regions for each hemisphere), subcortical volumes (bilateral caudate, putamen, pallidum, thalamus, cerebellar grey and white matter), MRPI and MRPI 2.0 values. Age, gender, education level and intracranial volume were also included in all ML models, but the feature importance (both in RF and XGB) showed that these variables were not relevant for classification, and the feature selection procedure excluded them from the final models. The hyperparameters of the two ML algorithms were tuned through five-fold cross-validation (fivefold cv) with randomized search (ten iterations) to maximize the accuracy [45, 46]. In detail, the dataset was split into K number of subset (folds) and the model was iteratively fitted K times, training it on (K-1) set and validating it on the Kth fold not used for training. The hyperparameters tuned for RF were: number of trees, features considered for splitting a node, levels in each decision tree, data points placed in a node before the node is split and points allowed in a leaf node. The hyperparameters tuned for XGB were: learning rate, maximum depth, minimum child weight, gamma and fraction of features to use. Further details on hyperparameters tuning in supplementary materials. The permutation feature importance (Mean Decrease in Accuracy, MDA) [47] was then evaluated, using 50 repetitions to ensure the reliability of the feature ranking, which might otherwise be biased by the multicollinearity among the training features. Feature selection was then applied by iteratively training the models on the variables ordered according to the permutation importance. Finally, the performance of the RF and XGB models trained on the most important features were evaluated using fivefold cv with 5 repetitions, and the mean and standard deviation of area under the curve (AUC), accuracy, sensitivity and specificity were calculated. A model was considered able to distinguish between groups when the mean AUC in the validation folders was > 0.85. The analyses were conducted with Python 3.9 and the packages scikit-learn v1.0.2.

Both PSP phenotypes had higher MRPI and MRPI 2.0 values than control subjects, and PSP-RS patients had significantly higher values than PSP-P patients (Tables 1 and 2). Both PSP groups also showed reduced thickness and volume in frontal lobe regions, but PSP-P had a more widespread cortical thinning, involving also the temporal and parietal lobes (Tables S1 and S2). This finding was confirmed by the direct comparison between the two PSP phenotypes, which showed cortical thinning in PSP-P patients compared to PSP-RS patients in several brain regions (Table S1). On the contrary, PSP-RS patients had a more severe atrophy of subcortical structures, including thalamus, pallidum and cerebellum (Table S3). Similar results were obtained in the early sub-cohort (Tables S4 and S5). The main differences respect to the whole cohort were that cortical involvement was detected only by thickness, while cortical volumes in the early sub-cohort were not different among the three groups, and that the cortical thinning in early PSP-P patients involved the frontal and parietal regions, sparing the temporal lobes (Table S4).

The MRPI had acceptable performance (AUC 0.88) and was superior to the MRPI 2.0 (AUC 0.81) in distinguishing between the two PSP phenotypes (Fig. 1 and Table S6). Similar performances were obtained in the early sub-cohort (Fig. 1 and Table S6). The ROC analysis identified optimal cut-off values of 16.25 for MRPI and 3.82 for MRPI 2.0 in distinguishing between PSP-RS and PSP-P (Table S6). The classification performances of MRPI and MRPI 2.0 in distinguishing PSP-RS and PSP-P from control subjects are described in supplementary materials and Table S6.

ML models with the MRPI and MRPI 2.0 used alone showed acceptable performance (AUC 0.86 and 0.79, respectively) in differentiating between PSP-RS and PSP-P patients, in line with ROC results. Lower performances were obtained by ML models using only cortical thickness (AUC 0.82), cortical volumes (AUC 0.78) or subcortical volumes (AUC 0.82), as shown in Tables 3 and 4 and Fig. 2. In most cases the performances were slightly higher using XGBoost than using Random Forest. ML models combining volumetric/cortical thickness data together with planimetric biomarkers (MRPI or MRPI 2.0) showed the highest classification performance in distinguishing the two PSP phenotypes, reaching mean AUC in the validation folds of 0.94 ± 0.04 using XGBoost and 0.91 ± 0.06 using RF (Table 5 and Figs. 2 and 3). In all these models, the MRPI was the selected feature with the highest importance score (Figs. 3 and 4). The Receiver Operating Characteristic (ROC) curve and the feature importance list of the best XGBoost and RF models are shown in Fig. 3. Of importance, similar results were obtained also in the differentiation between PSP-RS and PSP-P patients in the first years after disease onset (Figs. 2 and 4, Tables 5, S7 and S8), which is clinically more challenging. Classification performances of ML models in distinguishing PSP-RS and PSP-P from controls are described in supplementary materials and Tables 3, 4 and 5. All the hyperparameters of the best models are shown in supplementary materials and Table S9.

Finally, we investigated the performance of each structural MRI metric in distinguishing between early and late patients, separately for PSP-RS and PSP-P cohorts. As shown in Table S10, both classifiers (XGB and RF) showed that the cortical metrics were superior to the brainstem measurements (MRPI and MRPI 2.0) in distinguishing between early and late patients, with cortical thickness as the best feature in both PSP-RS and PSP-P cohorts. The main difference between the two PSP phenotypes was the higher performance of subcortical volumes in distinguishing between early and late patients in PSP-P cohort than in PSP-RS cohort.