The correlation between 9-HPT and patient-reported measures of upper limb function in multiple sclerosis: a systematic review and meta-analysis

We aimed to include all study designs of quantitative primary research that involved assessments with 9-HPT and an ULF PROM in people with MS, reporting correlation between the measures of their scores. In selecting the studies, we employed the following exclusion criteria: (1) studies with either objective outcome measures or PROMs alone, but not both; (2) studies with mixed populations; (3) non-peer‐reviewed publications. No limitations were put on date or language of publication.

To retrieve documents, we searched the following citation databases: Scopus, Web of Science, and PubMed. Reference lists reported in retrieved documents were checked to find additional potential eligible studies. Finally, where needed, we contacted the experts in the field. The following strategy was used to search Scopus: we generated a search query searching the title, abstract, and keywords record fields using two groups of keywords combined with an AND statement. The first group of keywords was intended to detect papers presenting results on sample of subjects with a MS diagnosis (i.e., multiple sclerosis, ms, spms, rrms, and ppms), while a second group of keywords were intended to detect papers including data on the 9-HPT (i.e., 9HPT, 9-HPT, nine hole peg test, nhpt, and 9-Hole Peg Test).

This query strategy was then adapted to generate the query string for use in the Web of Science and PubMed databases. The searches were conducted between December 2021 and June 2022. The used queries are reported in full in the Supplementary material.

Study selection was performed through Rayyan software independently by two authors. First, we inspected all records for duplicates identified by searching the databases. Next, the remaining records were screened for eligibility according to inclusion and exclusion criteria based on information reported in the title, abstract, and the full text of the manuscript. In case of disagreement in the articles selection, it was discussed by the two authors.

All selected papers were inspected for information including the year of publication of the paper, demographic and clinical information, the procedure use to perform the assessment of ULF, and effect sizes representing the association between 9-HPT and PROMs. Data were extracted from selected papers based on a predetermined coding sheet.

Regarding demographic information, the following information was retrieved about sample characteristic: distribution of age of (mean or median) and gender (number/percentage of PwMS by gender group). Clinical information about the sample extracted from the paper included information about distribution of MS disease course (number of PwMS per disease course), and disease duration (mean/median). Additionally, we collected information on the distribution of clinician-rated Expanded Disability Status Scale (EDSS; mean or median). The EDSS is a widely used ordinal measure quantifying disability and disease progression in subjects with multiple sclerosis (Kurtzke in 1983 [18]). EDSS scores derive from neurological examinations and range from 0 (normal neurological exam) to 10 (death due to MS) [18].

We also extracted information about the assessed ULF measures. We retrieved information about the scoring of 9-HPT, including whether the test was scored based on a single or both arms, and scoring metric (e.g., seconds); regarding the administered PROMs, including bibliographic information about the instrument, as well as the number of items included in the assessment.

Next, we extracted effect sizes representing the correlation between 9-HPT and PROMs. First, we retrieved information about the type of correlation used to evaluate the association between the 9-HPT and PROM scores (Pearson vs. Spearman). Note that PROMs may be scored either to indicate ULF, or reverse scored to indicate lack of ULF. In a similar way, the 9-HPT may be scored to reflect inability to perform the task (i.e., seconds required to complete the task) or ability (e.g., pegs per second). Depending on the scoring strategy used, a positive correlation may thus indicate convergence or divergence between the two measures in assessing ULF. For the purpose of the present study, when necessary, all correlations were recoded as positive when the reported correlation indicated convergence between the two measures, and recoded as negative when the correlation indicated a divergence between the measures. When we could not find correlations reported in the included studies (e.g., correlations were either not reported in full, or other effect sizes were reported but available information could not be used to obtain correlations), we contacted the first or corresponding authors asking them to provide us with missing correlations (i.e., authors were asked to compute correlations and provide us with the results). Study authors were contact by email five times over the course of 2 months.

Quality of the included studies was assessed by two independent critical appraisers (EG and RDG) using an adapted version of the Joanna Briggs Institute Critical Appraisal tools for cross‐sectional studies. More specifically, the appraisers scored each paper based on the following criteria: (1) Clear definition of subject inclusion criteria; (2) Detailed description of study subjects; (3) Use of valid and reliable measures for the assessment of the study outcomes; and the (4) Appropriateness of statistical analyses. For each criterion, appraisers rated “Yes” if the criterion was fully respected, “No” if not respected at all, “Unclear” if the criterion was partially respected, or applicability was uncertain. Each item was scored 1 for “Yes” responses, and 0 for either “No” or “Unclear” responses. Any disagreement was discussed and resolved by the two critical appraisers.

Because most of the studies provided more than one effect size computed on the same sample at one or more time points, thus resulting in a lack of independence of among the retrieved effect sizes within the same study, in estimating the meta-analytical correlation, we used a multilevel approach. More specifically, we implemented three-level meta-analytic model modeling three different variance components: sampling variance of the extracted effect sizes (i.e., the indeterminacy in effect sizes due to the use of samples, as opposed to population data to compute effect sizes); variance at the effect size level (i.e., within-cluster variance); variance at the study level (i.e., between-cluster variance). Note that in interpreting the magnitude of the meta-analytical correlation, we refer to the existing guidelines indicating correlations equal or above r =|0.10|, r =|0.30|, and r =|0.50| as reflecting, respectively, small, media, and strong effect sizes [19]. Heterogeneity of effect sizes was investigated by computing the Q test of heterogeneity, the I² statistic representing the proportion of true variation in observed effects, and by determining the percentage of heterogeneity due to the different variance components [20]. Note that Grubb's test was used to identify outliers prior to meta-analytical computations. Publication bias was investigated by inspecting the funnel plot of studies’ effect sizes against their relative standard error. Symmetry of the funnel plot was determined using a modified Egger’s intercept test [21]. More specifically, we fitted a multilevel model predicting study effect sizes with sampling standard errors (i.e., the square root of sampling variance) as a moderator: significance of the moderator effect would indicate a significant association between the standard error and effect size, indicating a potential “small-study” effect biasing our results. Classic fail-safe N was then used to evaluate the impact of a file-drawer problem (e.g., the number of unpublished studies reporting non-significant associations that would nullify emerging meta-analytical associations).

To determine the source of heterogeneity in effect sizes, we performed a series meta-regression analyses. First, we investigated whether the use of specific PROMs had an impact on the correlation with 9-HPT scores: to reach this aim, contrasts between specific PROMs were investigated using dummy coding. Note, however, that these contrasts were only examined if at least four non-independent studies per PROM were available [22]. Additionally, meta-regressions were performed separately for following variables: length of PRO questionnaire (i.e., number of items), type of correlation (Pearson = 1; Spearman = 0), coding of 9-HPT scores (seconds = 1; else = 0), source of 9-HPT score (Total score = 1; Single arm = 1), year of publication, and the following sample characteristics: mean/median age, prevalence of gender (% of female patients), disease course (% of RR patients), and overall severity of disability in the sample (EDSS ≥ 6.0 vs lower EDSS). Finally, we checked for the impact of study quality, using a median split on the overall quality score approach to distinguishing between high-to-moderate vs low quality (Quality score ≥ median = 1; else = 0). All analyses and visualization of results were performed using the metafor package for R [23, 24].

The PRISMA diagram in Fig. 1 provides a description of study selection flow. A total of n = 1049 records were retrieved by querying the databases (Scopus: n = 400; Web of Science: n = 332; PubMed: n = 317). First, we inspected all records for duplicates identified by searching the databases, resulting in the removal of n = 582 records. The remaining records (n = 467) were screened based on the information reported in the title, abstract, and the full text of the manuscript. This step led to identification of n = 22 eligible studies by both authors; while n = 18 studies were selected by only one of the authors and after disagreement discussion, they result in the further inclusion of n = 11 studies, leading to a total of 33 records. Additionally, n = 7 studies were included based on the inspection of study references [25‐31]. Hence, a total of n = 33 eligible studies were identified.

There were n = 21 [30‐50] studies that did not include information about the correlation between 9HTP and a validated PROM, but only their scores. After contacting the authors, missing information could be retrieved for n = 8 studies [32‐39]. Finally, we include in the review and meta-analysis a total of n = 27 studies [14, 25‐29, 32‐39, 51‐63].

The characteristics of included studied are reported in Table 1. The included studies involved a total of 3263 subjects, with a mean (or median) age ranging from 37.60 [26] to 58.2 [14], and generally a prevalence of female subjects, with an average of 66.8% females ranging from 50.0% [14] to 84.8% of the sample [62]. As regards disease course, RR patients were on average 63.4% of the sample, ranging from 0% [14, 39] to 100% of the sample [36, 58]; in turn, patients with either primary or secondary progressive MS were on average 36.6% of the samples. Please note that some of the studies did not report information about the distribution of disease course in the sample, and thus were excluded from these calculations (n = 4) [26, 32, 51, 52]. As regards EDSS, 16.0% of the studies reported either median or mean EDSS values ≤ 0.2.5, 60.0% of the studies reported either median or mean EDSS values in the 3.0–5.5 range, and 24.0% of the studies reported either median or mean EDSS values ≥ 6.0. Please note that n = 2 studies did not include information about the distribution of the EDSS in the sample [26, 32].

Concerning the studied outcomes, in most of the studies, the 9-HPT was only scored by recording the number of seconds required to move the pegs (n = 26) [25‐29, 32‐36, 38, 39, 51‐54, 57‐63], while a minority of studies used an alternative scoring based on the peg per second ratio (n = 3) [14, 37, 56]. Please note that we could only find one study using both these coding procedures [35].

Thirteen different ULF PROMS have been used among the included studies. Some of the PROMs assess upper limb ability in performing ADL, namely the ABILHAND-23 (n = 4) [32, 34, 39, 61], ABILHAND-26 (n = 1) [28], Manual Dexterity in Multiple Sclerosis adapted from Sunderland [64] (n = 2) [33, 55]), MAL (n = 2) [14–35, MAM-36 (n = 4) [29, 38, 56, 60], and NeuroQOL UE (n = 1) [58]. Other PROMs assess UL disability, namely the AMSQ (n = 5 studies) [27, 57, 59, 62, 63], DASH (n = 2) [26, 37], Duruoz's Hand Index [36], Upper extremity index (n = 1) [52], HAQUAMS’ Upper Mobility Subscale (n = 1) [25], Guy’s Neurological Disability Scale-Arms score (n = 1) [51], and the Performance scale-Hand Score (n = 2) [53, 54]). Finally, it is important to highlight that several PROMs are not validated for the MS population [33, 36, 52, 55], and others consist of either single items or multi-item scales included in PROMs assessing additional constructs beyond upper limb function [25, 51, 53, 54].

Results of evaluation of the quality of studies are reported in Table S1 of the Supplementary Material. Only n = 2 study [29, 35] reported the maximum score of 4, n = 4 studies [28, 37, 38, 59] reported a score of 3, n = 9 studies 14, [33, 39, 50, 53, 56, 57, 60, 63] reported a score of 2, n = 11 studies [25‐27, 32, 34, 36, 54, 55, 58, 61, 62] reported a score of 1, and n = 1 study [52] reported a score of 0. At a median value of 2 (Range 0–4), study quality was generally low-to-moderate. Overall, the majority of the selected studies reported adequate inclusion criteria (n = 18; 67%) and reliability and validity of UL measures (n = 16; 59%), while only a minority of studies provided in-depth information about study subjects and setting (n = 6; 22%) and rationale for using specific statistical analyses (n = 8; 30%). In particular, as regards the last criterion (i.e., adequacy of statistical analyses), most of the studies failed to report and/or discuss information on the distributional characteristics of study measures.

A forest plot of study effect sizes representing the correlation between 9-HPT scores and ULF PROM scores are shown in Fig. 2. Overall, we examined n = 75 distinct effect sizes reported in n = 27 studies. Grubbs test failed to identify outliers prior to meta-analytical computations. Overall, the central tendency analysis showed a strong association between 9-HPT scores and ULF PROMs scores (r = 0.51, 95% CI [0.44, 0.58]). The Q test for heterogeneity was significant (Q (74) = 515.68, p < 0.0001), indicating the presence of non-negligible heterogeneity among the effect sizes, but observed dispersion of effect sizes was mostly due to true heterogeneity (I² = 87.55). In particular, based on model decomposition of effect size variance, we saw that for all traits, most of the heterogeneity was due to variance at the study level (80.79%, between-cluster variance), followed by sampling variance (12.45%, variance due to sampling error), while variance at effect size level was the lowest (6.76%, within-cluster variance). Note that Grubb's test failed to identify outliers among the collected effect sizes.

The funnel plot of standard errors versus the study correlations was markedly asymmetric (see Fig. 3). However, contrary to the publication bias assumption (a positive correlation between effect size and its standard error), Egger’s tests was significant but a negative association was found between standard errors and correlations (b (73) = − 4.02 [− 5.10, − 2.94], p < 0.001), indicating that studies based on larger samples also tended to report stronger effect sizes. Additionally, the computed fail-safe N of 99,190 value was significantly larger than the recommended rule-of-thumb limit (5 × number of effect sizes + 10 = 385) [65]. These findings support the significance of the meta-analytic correlation emerging for the trait, ruling out the existence of a relevant publication bias problem.

Finally, we look at the results of moderator analyses. Estimated effects are reported in Table 2. A significant effect was found indicating a larger effect size for the association between PROM and 9-HPT when using the AMSQ questionnaire as opposed to the ABILHAND questionnaire (β = 0.212, p = 0.044). Note that contrasts between PROMs could only be examined between the AMSQ, MAM-36, and ABILHAND questionnaires due to the low number of studies identified for the other PROMs (n < 4). Finally, we found the effect size to be significantly larger (β = 0.186, p = 0.026) in studies with a mean or median EDSS level indicating severe disability (EDSS ≥ 6.0) compared with studies performed on samples with lower mean/median disability. No other significant moderation effect emerged.

On average, a strong correlation exists between 9-HPT scores and PROMs assessing ULF in ADL of patients with MS, supporting concurrent validity of both measures. However, the correlation does not come close to that expected for establishing equivalence of assessed constructs (e.g., r ≥ 0.8, [68]), thus indicating the two forms of measurement indeed assess different constructs. Results of the present study suggest that some questionnaire (i.e., AMSQ) may show a stronger convergence with 9-HPT scores than other instruments (i.e., ABILHAND), although these results should be taken with cautions due to the low number of studies included in the analysis. Finally, the size and the average disability of the recruited sample were found to affect the size of association between 9-HPT and ULF PROMs, such that the association tend to be stronger in large samples, and in those samples including larger groups of patients with severe disability along with less disabled patients.