Evaluating sensitivity and specificity of handheld point-of-care ultrasound testing for gynecologic pathology: a pilot study for use in low resource settings

Because this was a pilot study, sample size was not powered for statistical significance. The first twenty sequentially-enrolled subjects underwent clinician-performed pelvic POC-US with each of the 3 units (GE Vscan (Vscan), Sonosite Iviz + L38v linear probe (Iviz), Philips Lumify C5-2 curvilinear probe + Samsung Galaxy Tab S2 9.7 with mobile app (Lumify)) in rotating order followed by diagnostic pelvic ultrasound (Dx-US) using one of the following machines: Philips IU22 (Koninklijke Philips N.V, Amsterdam, Netherlands), Medison Accuvix 20 (Samsung Medison, Seoul, South Korea), Voluson E10 (GE Healthcare Ultrasound, Milwaukee, WI, USA) with 4–5 MHz abdominal probe. Clinician-performed POC-US was performed by single author (MT), a third-year resident in Obstetrics and Gynecology, with supervision by RDMS certified sonographer and co-investigator (KS). POC-US was performed prior to reference imaging, therefore reference test results were not available to investigators at the time of POC-US. A diagnostic ultrasound (Dx-US) in accordance with AIUM practice parameters for performance of an ultrasound examination of the female pelvis was then performed by a different OBGYN sonographer (RDMS) who was blinded to any findings on the POC-US [16].

Test images from each POC-US were blindly compared to Dx-US images by two independent sonographer reviewers. Reviewers completed a qualitative assessment for each test image of POC-US image quality and for correlation of POC-US image to Dx-US image by non-validated 5-point Likert scale from 0–5 (0—structure not seen, 1—poor to 5—excellent) for each test image. The following gynecologic structures were scored: uterus, endometrium, cervix, left ovary, right ovary, cul de sac, pathology of interest. Gynecologic structures were considered satisfactorily assessed by a POC-US device if the mean sonographer review score was ≥ 3 for image quality score and ≥ 3 for correlation score. Collapsing Likert scale response into dichotomous neutral/positive (score 3, 4 or 5) or negative (score 1, 2) categories performed to minimize ambiguity and clarify intent of responder to better capture trends in data. Neutral response aggregated with positive response to reduce response bias related to survey satisficing and maintain focus on lack of negative score as primary outcome of interest [17]. Aggregate score for each device was calculated by totaling the number of scans possessing qualitative assessment scores ≥ 3 for all structures. Two-tailed Fisher’s exact tests used to compare proportions of scans with assessment scores ≥ 3 between devices to inform superiority of a single POC-US device.

Qualitative measurements of gynecologic structures by each POC-US device were compared to measurements obtained by Dx-US for uterine volume, endometrial thickness, left and right ovary mean diameter, and pathologic structure mean diameter. For both POC-US and Dx-US, uterine volume was calculated by measuring the maximum length (excluding the cervical component), anteroposterior and transverse diameters of the uterine corpus, and using the formula for the volume of a prolate ellipsoid: V = 0.52 × (L × AP × T) [18]. Endometrial thickness was measured at the thickest part of the endometrium perpendicular to its longitudinal plane in the anteroposterior diameter from echogenic to echogenic border. Ovaries were measured in 3 dimensions (longitudinal, transverse, and anteroposterior diameters) on views obtained in 2 orthogonal planes [16]. Pathologic structures included simple or complex adnexal cysts or myomas. Simple ovarian cysts were defined as > 3 cm, thin and smooth walled, round or oval, anechoic spaces with no flow by means of color Doppler US [19]. Paraovarian and paratubal cysts were considered together with ovarian cysts. Simple ovarian cysts were measured in 3 dimensions (longitudinal, transverse, and anteroposterior diameters) on views obtained in 2 orthogonal planes. Cysts of any size which contained septations, solid or mixed cystic/solid components and were considered complex ovarian cysts and were measured as described above. Myomas were measured in three perpendicular diameters [18]. Endometrial pathology defined as thickened endometrium > 4 mm in a post-menopausal subject [20], fluid filled endometrium or IUD in situ. Endometrial measurements > 4 mm in a pre-menopausal subject were not considered pathology. Absolute mean difference in measurements defined as absolute difference in measurement by POC-US minus Dx-US. P < 0.05 for absolute mean difference considered significant “disagreement” in measurements and one-way analysis of variance performed to inform superiority of a single POC-US device.

A single best POC-US device was then selected by a combination of (1) highest aggregate score for image quality > 3, (2) highest aggregate score for image correlation > 3, and (3) lowest number of “disagreements” in quantitative measurements of individual structures (as defined above), and (4) suitability for field use based on total continuous scan time on one battery charge and ease of use.

The selected device then underwent further testing with additional prospective enrollment of forty subjects in a sequential convenience sample. Because this was a pilot study, sample size was not powered for significance, though a larger sample size of 40 was chosen to allow for inclusion of subjects with a variety of gynecologic pathologies, each of which has a baseline low prevalence. Clinician performed POC-US with selected device was performed by single blinded operator (MT) and Dx-US by RDMS. POC-US was again performed prior to reference imaging, therefore results of reference imaging were not available to POC-US operator at the time of image acquisition or interpretation. Similarly, results of POC-US were unknown to RDMS at the time of reference imaging. Interpretation of test images for diagnosis was performed by clinician POC-US operator (MT) at the time of image acquisition. Interpretation of reference imaging for diagnosis was by board-certified maternal–fetal medicine specialists who were blind to results of POC-US. At this research institution, maternal–fetal medicine specialists are also certified by AIUM for interpreting gynecologic ultrasound studies. All indeterminate or missing results from the POC-US were considered false-negative (in those with pathology present on reference imaging) or false-positive (in those with pathology absent on reference imaging).

A pilot field study of POC-US in a low-resource setting was performed in the remote and mountainous Borgne community on the north coast of Haiti, west of Cap Haitian. The region is accessible only by rough footpaths. The majority of the population lives in extreme poverty and relies on subsistence agriculture, fishing and trade of crops at local/regional markets. There is a low level of educational attainment and limited access to clean water or sanitation and no current access to diagnostic imaging services.

For this prospective field study, 20 total subjects were enrolled in a sequential convenience sample. Because this was a pilot study, sample size was not powered for statistical significance. Subjects were women presenting to a mobile health clinic for any complaint of possible gynecologic etiology. Prior assessment of the perceived women’s health needs in the mobile clinics revealed a need for improved general gynecologic care for benign and routine pathologies, most commonly abnormal uterine bleeding and pelvic pain. Women were approached and enrolled in January, 2018 with the help of Haitian Creole translator. Women were excluded if they were less than age 18 at the time of the study or currently pregnant. Subjects did not receive compensation or incentives for participation. Clinician performed POC-US was performed and interpreted by single author (MT) with supervision by co-investigator (NW) after patient consent.

Because there is no access to imaging capabilities in the region of the field study, no reference imaging was performed for comparison to POC-US imaging for subjects in the field study.

Demographic characteristics of the two groups were compared with chi square tests for categorical variables. Distribution of continuous demographic data was established using the Shapiro–Wilk test followed by Mann–Whitney U test for non-parametric data.

Two-tailed Fisher’s exact tests were used to compare proportions of scans with assessment scores ≥ 3 between the three test devices to inform superiority of a single POC-US device. Absolute mean difference in measurements of gynecologic structures (defined as absolute difference in measurement by POC-US minus Dx-US) compared with one-way analysis of variance to inform superiority of a single POC-US device.

Continuous variables compared by linear regression with a value close to 1 representing high agreement between device measurements. Bland–Altman plots were constructed as a visual representation of the mean difference between single paired measurements by the two methods. The limits of agreement indicated by the dotted lines and calculated as the interval of two standard deviations of the measurement differences on either side of the mean difference (solid line). A priori determination of acceptable limits of agreement was not established. Agreement in measurements defined as 95% of data points (difference between the two measurements) included in the 95% limits of agreement. Sensitivity, specificity, positive predictive value and negative predictive value were also calculated by 2 × 2 contingency table for 1) dichotomous outcome of presence or absence of pathology and 2) stratified by individual diagnoses. All statistical analyses were performed using IBM SPSS Statistics 21.

Of the 20 total participants who underwent POC-US assessment for device comparison, reference imaging demonstrated 4 subjects with myomas, 8 subjects with adnexal pathology (2 complex and 6 simple ovarian cysts), and 6 subjects with endometrial pathology (4 with IUD in situ, 2 with thickened endometrium).

Satisfactory assessment of gynecologic structure as defined by mean image quality score ≥ 3/5 and correlation score ≥ 3/5 occurred most frequently using Iviz, then Lumify and then Vscan (Fig. 2).

Image quality scores did not statistically significantly differ between devices for cervix, right ovary or left ovary. There were statistically significant differences between POC-US devices for uterus, endometrium, cul-de-sac, pathology of interest, and aggregate score (Fig. 2a). The Iviz and Lumify had statistically higher number of scans with image quality scored ≥ 3/5 as compared to the Vscan for uterus, cul-de-sac, pathology of interest, and aggregate image quality. The Iviz had statistically higher number of scans with image quality scored ≥ 3/5 as compared to the Lumify for uterus, endometrium, and pathology of interest (Fig. 2a). Compared to the Vscan, Iviz and Lumify were both statistically significantly more likely to have an aggregate image quality score ≥ 3/5 (66% Iviz versus 39% Vscan, P < 0.01; 59% Lumify versus 39% Vscan, P < 0.01) with no significant difference between Iviz and Lumify.

Image correlation scores did not statistically significantly differ between devices for uterus, endometrium, cervix, left ovary and right ovary. There were statistically significant differences between groups for image correlation score for cul-de-sac, pathology of interest, and aggregate (Fig. 2b). The Iviz device had a statistically significantly higher aggregate number of scans with image correlation scored ≥ 3/5 as compared to Lumify or Vscan (33% Vscan vs. 55% Iviz, P < 0.01; 45% Lumify vs. 55% Iviz, P < 0.01), with no difference between Vscan and Lumify.

On quantitative device comparison, the absolute mean difference in measurements between diagnostic ultrasound and each of three devices did not have statistically significant disagreement for endometrial thickness, uterine volume, left ovarian diameter and pathologic structure mean diameter (Fig. 3). The opposite finding was demonstrated for measurements of right mean ovarian diameter which had significant disagreement in measurements by POC-US as compared to Dx-Us (Fig. 3). There were no statistically significant differences between devices for any measurement as determined by one-way analysis of variance (Fig. 3).

Based on these data, all three devices performed equivalently on quantitative device comparison, though there was trend towards decreased mean error in measurements and narrower limits of agreement for both Iviz and Lumify as compared to Vscan. The Iviz and Lumify performed overall equivalently and Vscan device inferiorly in qualitative measurements. Ultimately, Lumify was selected based on superior overall ease of use and applicability to low-resource setting in regarding to battery life, portability, and cost. In addition, Lumify was found to be superior in ability to adjust depth and gain.

Between October 1 and 31, 2017, 40 patients were assessed for initial eligibility and invited to participate. “Appendix 2” shows the flow of patients through the study, along with the primary outcome of gynecologist performed and interpreted POC-US of gynecologic pathology. Patients who were excluded (and reasons for this) or who withdrew from the study are noted. In total, 40 patients completed the study, a completion rate of 100%. Subjects with more than one diagnosis were included in multiple categories therefore total number of included test subjects was 110.

Of 40 total participants who underwent POC-US with Lumify, reference imaging showed 10 subjects with myomas (mean diameter 3.9 cm), 5 subjects with adnexal pathology [4 with simple cysts (mean diameter 4.9 cm) and 1 hemorrhagic cyst (mean diameter 2.3 cm)], 12 subjects with physiologic follicular cysts (mean diameter 1.7 cm) and 10 subjects with endometrial pathology [6 subjects with IUD in situ, 3 subjects with post-menopausal thickened endometrium (mean thickness 0.7 cm)].

In validation phase, the Lumify had a high overall statistical concordance with Dx-US in operator-interpreted diagnosis of normal pelvic anatomic structures as well as pelvic pathology as assessed by Cohen’s unweighted kappa (Table 2). There was almost perfect agreement in assessing presence of endometrial pathology and type of endometrial pathology (κ = 0.940, 0.841, respectively P < 0.01). There was substantial agreement in assessing uterine position, presence of fibroids, presence of adnexal pathology, type of adnexal pathology, and final diagnosis (κ = 0.752, 0.731, 0.701, 0.716, 0.761 respectively, P < 0.01). There was fair agreement between in ability to assess right ovary and left ovary (κ = 0.357, 0.415, P = 0.022 and P < 0.01 respectively).

Lumify measurements of all continuous variables (endometrial thickness, uterine volume, left and right ovarian mean diameter, adnexal pathology mean diameter and myoma mean diameter) showed no statistical disagreement from Dx-Us by linear regression (Fig. 4i). Bland–Altman plots were constructed with > 95% of data points falling within the 95% limits of agreement. Bland–Altman bias ranged 0.29 to 31.5 for these measurements (Fig. 4ii). Overall the sensitivity, specificity, positive and negative predictive value of the Lumify POCUS was excellent with a specificity and sensitivity 80–100% for detecting myoma, endometrial and adnexal pathologies, and with PPV and NPV ranging 76–100% (Table 3). Contingency tables for accuracy of POC-US are available in Additional file 1: Table S1.

Demographic characteristics of subjects are shown in Table 1. As compared to subjects in the United States, those in the field study were majority Black race (100% vs. 28%, P < 0.01) and not overweight or obese (100% vs. 38%, P < 0.01). Pain was the single given indication for pelvic ultrasound in this population (100% vs. 17%, P < 0.01). Of 20 total participants, the POC-US identified normal gynecologic anatomy in 90% of subjects, although ability to identify ovaries was limited by transabdominal approach. Only 1 subject was identified as having leiomyomata and 1 subject had an ovarian follicular cyst.

The Vscan (GE) ultrasound equipment was previously purchased by the institution and used on temporary loan from the Department of Obstetrics and Gynecology at University of Rochester for the study period. Sonosite supplied Iviz equipment on temporary loan for the study period at the request of one author (MT). Lumify equipment was rented from Philips for the study period. The authors and University report no conflicts of interest regarding this equipment. None of the authors are or were in any contractual agreement with GE, Sonosite, Philips, Samsung regarding the study equipment (other than purchasing/rental/loan agreements) and none serve as speakers/experts for any of the companies. Neither the authors nor the University received any payment, support, or benefits in relation to this study from these companies (other than equipment loan as noted above). GE, Sonosite, Philips, and Samsung did not have any involvement in development of this study, the analysis or review of the data, writing of the manuscript, and did not have any approval or decision making in the submission of this manuscript.