Automated segmentation of long and short axis DENSE cardiovascular magnetic resonance for myocardial strain analysis using spatio-temporal convolutional neural networks

Myocardial deformation holds important diagnosis and prognosis value for the assessment of heart conditions [45], which remains one of the largest causes of death worldwide [5]. In particular, myocardial strain provides significant added value compared to the left ventricular (LV) ejection fraction (LVEF) [41, 52]. LVEF is a ubiquitous biomarker for the assessment of cardiac conditions, but is limiting when we need to understand underlying mechanisms, and cannot provide regional information [2, 30, 57]. A normal LVEF can mask cardiac dysfunction and lead to missed or delayed detection of disease. Myocardial strain has been shown to be particularly useful for the diagnosis of congenital heart diseases [9] and prediction of cardiac function after repair [24, 53], while severe regional strain impairment correlates with myocardial infarction (MI; [10, 34, 35]. Strain assessment also has added value for diagnosis [15, 36, 46] and prognosis [38] of cardiomyopathies, and is being recommended for cardiotoxicity analysis after cancer treatment [17, 40, 51, 55]. More recently, the assessment of myocardial strain is recommended in international guidelines for the care of cardio-oncology patients [32].

While myocardial strain can be estimated from a range of modalities and acquisition protocols, cardiovascular magnetic resonance (CMR), and in particular cine displacement encoding with stimulated echoes (DENSE) imaging, is a promising technique for strain assessment [1, 29, 64], and has been shown highly reproducible [60]. The adoption of echocardiography-based methods, like tissue Doppler imaging and speckle tracking, can be limited due to the need for highly trained operators [8]. Other CMR techniques, like tissue tagging and phase-velocity encoding, have also shown some limitations in their ability to provide accurate strain estimation with high spatial resolution [1, 20]. Feature tracking CMR using standard cines does not reliably quantify regional strain [56].

The application of DENSE to clinical pipelines is limited by the lack of automation for the extraction of myocardial strain. Current tools like DENSEanalysis [12, 47] require significant human interaction in contouring the LV epicardial and endocardial borders. In DENSE images, displacement information is encoded in the image phase, and while the most advanced algorithms are able to automatically unwrap the phase and extract Lagrangian strain from pixel displacements, these steps generally still require human adjustments, either to select specific hyper-parameters or guide phase-unwrapping by selecting seed points in non-wrapped areas of the phase.

The use of deep learning (DL) for cardiovascular imaging, and in particular convolutional neural networks, has been significantly increasing in the past years. These models are being used extensively for CMR segmentation [22, 28, 42, 43, 61, 61]. Recently, nnU-Net was presented by Isensee et al. [21] which builds on U-Net, the segmentation model developed by Ronneberger et al. [44] in 2015. Strategies were developed in nnU-Net to improve the performance of U-Net for a highly diverse range of medical image segmentation tasks, and it has been widely adopted as a benchmark method. nnU-Net optimizes some hyper-parameters used in training U-Net models, either empirically, or on a task-by-task basis given the characteristics of a given dataset. Data augmentation is also performed with a wide range of transformations, while the inference performance is boosted by implementing test-time augmentation strategies. Additionally, there is an emergence of DL methods for automated cardiac motion analysis [4, 31, 43, 59]. However, there are few studies related to DENSE imaging for post processing. Recently, Ghadimi et al. [11] showed, using basic 2D U-Net models, very promising results for automatic segmentation and extraction of strain from short-axis DENSE acquisitions. However, these 2D method processed each frame independently, so temporal coherence, T1 relaxation and blood flow through the slice were not incorporated, which can make segmentation of the first and last frames difficult due to lack of contrast. Also, only short-axis analyses were performed. Kar et al. [27] showed how this could be achieved using 2D deep CNNs, but exhibit the same limitations.

In this study, we explore the use of DL to automate the segmentation of LV myocardial short and long-axis DENSE images. In particular, we show how temporal redundancies in cine sequences (2D + time) can be leveraged to improve segmentation performance and coherence. The nnU-Net pipeline was extended in this work under the MONAI framework [37] to create models that can easily be integrated into scanner platforms and clinical pipelines. Models are validated by comparing segmentation maps to ground-truth manual contouring, both in internal and external test datasets. Further validation is performed by automatically extracting Lagrangian displacement and strain values, and quantifying variability in relation to previous studies [3]. To this purpose, we extended the DENSEanalysis Matlab tool ([12], Mathworks, Natick, Massachusetts, USA) to be able to automatically analyze studies with minimal interaction using the automated contours as input.

We trained 2D and 2D + time nnU-Net models to segment the LV myocardium from DENSE magnitude images, training separate models for SAx and LAx datasets. 2D + time architectures were found to leverage temporal redundancies and remove segmentation artifacts over more simplistic 2D approaches. We re-implemented and simplified the nnU-Net framework using the MONAI library, based on the DynU-Net architecture. 2D + time DynU-Net models were successfully validated as they showed, when used in a complete pipeline to calculate Lagrangian strain, good agreement with manual analysis. The DENSEanalysis MATLAB tool was extended to automatically analyze full datasets with reduced processing time and minimal user interaction (from several minutes of manual processing, excluding segmentation, to less than 30 s). The modification of DENSEanalysis is a critical aspect of this study, as it can be used for further studies, even for standard manual processing. The generalizability of the models was validated by analyzing a second test set acquired from a different center and scan-rescan variability was quantified in multiple scanners.

In our work, we noticed that training 2D architectures based on individual frames only was sub-optimal. With cine DENSE, frames have decreasing contrast and brightness at later times in the cardiac cycle due to T1 recovery. While the inherent variability from this loss of contrast makes it harder for a network to perform well, its impact can be mitigated by implementing data augmentation with intensity-based transforms. However, the effects of other artifacts cannot be mitigated in this way. Early cine DENSE frames have poor contrast between blood and myocardium as the blood has not yet left the imaging slice and spiral streaking artifacts (sometimes rather substantial) may also be present in these frames. As a consequence, 2D networks frequently fail in the initial frames (see Fig. 3). Additionally, there is a significant reduction in myocardial signal over the cardiac cycle (p \(<{10}^{-67}\) between mean LV myocardial signal in frame 1 vs mid-frame, and p \(<{10}^{-113}\) for mid frame vs last frame), which further motivates the use of temporal networks.

Using 2D + time architectures enables the networks to leverage temporal redundancies in the image series. As we can see in Table 4, this first leads to a slight increase in average performance. More interestingly, we noticed that the cases noted above, where 2D segmentation fails on the early frames, are generally well segmented by the 2D + t architectures, as can be seen in Fig. 3. This trend over the cardiac cycle is shown in Fig. 2. While the blood-to-myocardium contrast is poor as expressed above, the 2D + time nnU-Net model is able to more accurately predict segmentation labels than its 2D counterpart.

This idea could be explored further. While we saw that temporal redundancies in a series of 2D images can be modeled by 3D convolutional architectures, these types of models might not be the best suited as the models are forced to learn temporal and spatial context using 3D convolutional kernels. Spatio-temporal relationships might be better modeled by other types of architectures, for example recurrent-based (RNN), by incorporating convolutional long short-term memory (LSTM) units in U-Net architectures as done by Lu et al. [31].

nnU-Net is an extensive framework that is currently the benchmark for numerous medical imaging tasks. It has been used in various challenges [13, 21‐23, 58] was explored widely for cardiac segmentation [18, 19], and is frequently used as a base model in recent papers [6, 62]. However, nnU-Net has become highly complex with successive modifications, and shows limited flexibility for customization. Reproducing the entire nnU-Net pipeline in external software platforms, particularly on scanners where Python code is not normally supported, becomes challenging. Our MONAI implementation drastically simplifies the translation process, while showing results that are on par with nnU-Net. In addition, thanks to early stopping and caching options available in MONAI, we reduced the training time significantly.

We showed that the strain values calculated from DENSEanalysis were generally in excellent agreement when obtained from the manual ground-truth and the automated LV myocardium segmentation, particularly reproducing the Ecc transmural gradient in SAx views reported by Zhong et al. [64]. Additionally, when we compare to the results obtained from the recent multi-center reproducibility study [3], we can see that the strain agreements between the automated and manual segmentations are on par with the intra-user variability, which showed an excellent ICC across all strain components (0.93, 0.94, 0.92 resp. for Ecc, Ell, Err), and an excellent CoV for Ecc and Ell reproducibility but only fair for Err (resp. 3.0, 2.9 and 24.6). We can conclude, based on the strain comparability results on independent test sets, that the 2D + time DynU-Net inference does not induce a significant variability in the strain calculations. The quality of segmentation still plays an important part in the ability to calculate strain. For a limited number of cases, it happened that the contours produced at inference time by the trained 2D + time DynU-Net were not continuous, which made it impossible to analyze with DENSEanalysis to extract strain values (4 SAx and 3 LAx slices had this problem in the independent test set). This generally happens when the myocardium is particularly thin, especially in the LAx apical regions, or when the image contrast is reduced. From the perspective of doing inline quality control, when such a network will be used directly on the scanner, it is critical that the topology of the myocardial segmentation is preserved. To guide models in that respect, previous works have shown interesting results when topological priors are enforced during training time [7]. The impact of topological errors on the rest of the pipeline (phase-unwrapping and strain extraction) will however be mitigated when training additional models to automatically process these steps as well in future work.

We validated the trained DynU-Net architecture further on a second test set, from a different center, different scanners, and with different acquisition parameters. Despite these differences, the segmentation performance is on par with that found in the original test set, which is impressive given the domain shift. However, the most adverse cases, where Dice score and/or Hausdorff distance is impaired, show generally worse performance than those from the original test set, up to a point where segmentation can extend outside of the myocardium (Additional file 1: Fig. S2). More importantly, we see cases where the topology of the myocardium is lost, making the following strain analysis with DENSEanalysis impossible. Four of 45 SAx exams could not be processed because of these topological failures.

Additional work needs to be done to preserve the topological structure of the myocardial segmentation, as seen in the previous section, and this is particularly important to successfully conduct LAx analyses. We applied the long-axis 3D DynU-Net model on the second test set as well. While this resulted in reasonable segmentation performance (Dice score of 0.81 ± 0.04, Hausdorff distance of 14.5 ± 10.5), a majority of these cases showed a non-continuous topology, which made it impossible to conduct a strain analysis.

These segmentation issues mainly come from the distribution shift between the secondary test set and the original dataset. This shift is substantial, given that the data was acquired in a different center, on different scanners, and with different acquisition parameters. In particular, observers segmented a thicker myocardium as ground-truth compared to the original dataset, and the DENSE protocol typically acquired cardiac frames finishing just after peak systole, rather than filling the whole cardiac cycle. While the trained DynU-Net models are effective on the second test set, the distribution shift does impact performance, and domain adaptation strategies will prove necessary if we want to successfully export such networks to multiple centers. While intra- and inter-scanner variability are generally better with manual analysis, the variability introduced by the automated myocardial segmentation for short-axis data remains within reasonable limits. Indeed, the Bland–Altman reproducibility metrics (bias and limits of agreement) are either on par or superior to the ones exposed in the multi-center reproducibility study [3] for inter-observer variability. This is even more remarkable given the domain shift described above, which shows how promising 2D + time U-Net-based models can be for DENSE image analysis.

While the datasets used in this study are diverse, including subjects with various conditions, acquired and analyzed at different times, the quality of the ground-truth manual data was imperfect in many cases due to the limited amount of time available for the analyzing researcher to perform the manual analysis, as well as systematic differences between observers. In addition, the main dataset that we used to train our models was acquired from a single center on two different 3T scanners only. While we showed that, in terms of raw segmentation metrics, our models are able to mitigate the distribution shift that is introduced when using them for inference on external data, some topological problems remain and are necessary to be addressed if we want to successfully follow segmentation with strain extraction. Training on more diverse datasets, using domain adaptation solutions [14, 25, 39, 54] or enforcing topological coherence are potential strategies to mitigate this issue [7].

In this work, we explored how including temporal information in the DL models results in more robust segmentations of DENSE images than 2D models. However, DL was not applied to the rest of the pipeline, where minimal user interaction remains. In particular, current methods for phase unwrapping and strain estimation suffer from regularization and partial volume effects. In future work, we will extend the architectures developed in this study to understand how they might improve the processing of phase unwrapping and strain estimation, combined with the segmentation work presented in this study that will help future strain analysis. Ghadimi et al. [11] and Kar et al. [26] recently explored the use of 2D CNNs for automating the unwrapping of the myocardial phase, removing the need for seed points. It will be interesting to understand how temporal models can make robust end-to-end automated pipelines.

Finally, the development work that we have done on the models and DL pipeline was made with the aim of simplifying the portability of the process, especially for implementation on the scanner, for scan-time provision of strain maps. However, the actual impact on clinical pipelines is yet to be validated. In the future, we plan to test our models in inline settings and validate their use on clinical data.

In this work, we trained DL networks to segment the LV myocardium from both SAx and LAx DENSE images for the first time, automating the most time-consuming step in the DENSE analysis. Temporal U-Net-based models show excellent robustness to contrast variability and acquisition artifacts when compared to manual methods, with no impairment of strain calculation if used in conventional DENSE analysis pipelines.

DL is extremely promising for the rapid and accurate extraction of cardiac strain information. Temporal models move the processing of DENSE sequences one step closer to the clinical setting, to estimate myocardial strain for patients suffering from cardiovascular diseases at the scanner, reducing the need for burdensome and time-consuming processing tasks, and improving the patient journey.