Automated assessment of the smoothness of retinal layers in optical coherence tomography images using a machine learning algorithm

The dataset for this study was supplied by the Farabi Eye Hospital at Tehran University of Medical Sciences in Tehran, Iran. The RTVue XR 100 Avanti device (Optovue, Inc., Fremont, CA, USA) was used to capture enhanced-depth imaging optical coherence tomography (EDI-OCT) images between 8:00 and 12:00 a.m. Patients were positioned appropriately, and 8 mm * 12 mm raster patterns of OCT B-scans were captured. For image analysis and SI calculation, scans passing through the fovea as well as equally spaced rasters in the superior and inferior macular areas were chosen. In OCT scans above or below the fovea, the IPL and OPL layers are often separated and parallel. But, due to the near proximity of IPL and OPL in the foveal region, it might be challenging for the algorithm to differentiate between the layers.

To evaluate the accuracy of the algorithm in a robust and precise manner in the current investigation, we selected similar OCT rasters in all patients with equal distances from the fovea and assessed the accuracy of segmentation and measurement.

We omitted low-quality scans due to conditions like cataract or other causes of hazy media in which the IPL and OPL were not discernible. Due to structural OCT alterations, such as the presence of exudate, which may impact OPL and IPL segmentation, we also removed some OCT scans.

Finally, we included 50 OCT B-scans from 21 patients with various stages of diabetic retinopathy (mild to severe non-proliferative diabetic retinopathy and proliferative diabetic retinopathy) in this study. The images were originally 256 × 728 pixels in size. The inner plexiform layer (IPL) and outer plexiform layer (OPL) in OCT images were manually segmented by two retina experts (H.R.E and E.K.P using the public domain ImageJ software (available in the public domain at Fiji http://​imagej.​nih.​gov/​ijwas).

The proposed automated algorithm, developed in the current study by Python framework, consists of three stages: pre-processing; segmentation of the retinal IPL and OPL layers using support vector regression (SVR); and smoothness index calculation. The proposed steps of the automated method are depicted in Fig. 1. After denoising the original image (Fig. 1a) by non-local means denoising algorithm (Fig. 1b), the retinal area is localized using Gabor filters (Fig. 1c). A search strategy is used to perform the initial prediction of the IPL and OPL layers (Fig. 1d). Then, as morphological operations, dilation and erosion are performed in order to remove undesirable areas and improve the predicted layers (Fig. 1e). Finally, the boundaries are detected using an SVR-based model (Fig. 1f).

In preprocessing stage, low pass Gaussian filter with kernel size of 11 × 11 as well as a non-local means algorithm were used to reduce the noise of OCT images. This algorithm is an image noise reduction method that, unlike local and global methods, uses a different and smarter approach for denoising in such a way that it considers similar parts in image in terms of texture and structure, and detects and removes noisy points by averaging. Then, histogram equalization was applied to increase the contrast of the image.

Localization of retinal area was carried out using Gabor filter. Upper and lower limits of the retina were obtained using segmentation of internal limiting membrane (ILM) and retinal pigment epithelium (RPE) layers. Gabor filtering with two different kernels was used for obtaining these boundaries as shown in the following formula.whereand \(\lambda\) is the wavelength of sinusoidal component, \(\theta\) represents the orientation of the normal to the parallel stripes of Gabor function, \(\psi\) denotes the phase offset of the sinusoidal function, \(\sigma\) is the standard deviation of the Gaussian envelope and \(\gamma\) is the spatial aspect ratio and specifies the elasticity support of Gabor function. By properly selecting of \(\lambda ,\theta ,\psi ,\sigma\) and \(\gamma\) parameters, we obtained two different kernels that shown in Fig. 2(1,2). After applying them to the original image (Fig. 2a), the boundaries of the ILM and RPE were obtained that shown in Fig. 2b. In this way, as you can see in Fig. 2c the retinal area is separated from the rest of the image. The selected parameters for two kernels were summarized in Table 1. In the current study, the initial parameters were chosen in a way that the resulted segmentation visually resembles the output of the original OCT image. In this manner, we find a ground truth interval for the parameters. In the next step the values were carefully changed to increase the precision of the obtained layer according to the original image.

We automatically acquired the retinal area as a region of interest in the previous steps, which has the benefit of reducing background noise and therefore facilitating the segmentation of the IPL and OPL layers. For segmentation of IPL and OPL layers, we smooth our original image (Fig. 3a) using Gaussian filter (Fig. 3b) and then go to reduce image noise (Fig. 3c). In the next step, we go to the enhanced image using histogram equalization that result of this process is shown in Fig. 3d. Initial prediction of the layers is obtained by employing a search strategy based on similarity and correlation between intensity of the pixels in these layers. According to the separability of the image histogram, after some mentioned preprocessing steps, conventional thresholding algorithms can be used to find the optimal threshold in a simple and achievable manner. The points on our target layers (IPL and OPL) have a brightness above 30, while other points reflect a brightness below 15, as can be observed after applying the Gabor filter. Testing different brightness thresholds between 15 and 30, one observes that the output images have no significant difference, so the threshold of brightness is set to 20 in the current study. After setting a suitable threshold between foreground (IPL and OPL) and background (the rest of the image) pixels, a binary image was created using global thresholding algorithm, this process is generally shown in Fig. 0.3. White pixels indicate estimated IPL and OPL locations in the generated binary images (Fig. 3e). Then, morphological operators are employed to detect the desired layers, erosion eliminates the individual white pixels which is not connected to any meaningful layer and dilation helps to get a connected and continuous layer. Also, we need to perform a connected-component labeling for eliminating any extra layer, such layers may occur in the process like the one which appears in the upper right corner of Fig. 3e. To be more precise, the process begins with two times erosion, followed by a dilation. Then it uses a connected component operator followed by an erosion and a dilation. The continuous boundaries were created and outliers were removed as depicted in Fig. 3f [40]. In order to skeletonize and get a curve-shape for IPL and OPL identifiers, we create two separate lists of pixels, the first is according to the far most upper white pixels in a column considered as IPL and the second stores the lowest white pixels in each column considered as OPL. It is possible for these two lists to have common points, it occurs whenever the IPL and OPL stick together in the original OCT slabs, in these cases this common pixels appear in both IPL and OPL lists, separately.

In the final step, due to discontinuity of pixels in the resulted lists, we need approximation methods for filling the gaps, that we will describe in the following sections.

A modeling-based technique was utilized to achieve precise segmentation of the IPL and OPL layers. Two approaches were investigated for this purpose: classic and machine learning-based methods. In the classic technique, cubic spline interpolation is employed to determine the exact target layers’ (IPL and OPL) locations. Cubic spline interpolation is a method for modeling objects with curved shapes [41]. On OCT slabs with relatively regular and organized layers, this approach produces satisfactory results (Fig. 4c). However, as seen in Fig. 4, it fails in OCT slabs with relatively disorganized layers (Fig. 4f).

To address this issue, we presented a machine learning technique based on support vector regression (SVR). SVR, as a supervised-learning technique, works utilizing points on layers and a prediction function. When the labelling is wrong, the model predicts and replaces the missing values, which is an advantage of employing SVR.

Since our model was a nonlinear function, the data was transformed into a higher-dimensional space known as kernel [42]. SVR can be achieved by using various kernels such as polynomials [43], sigmoid [44], Radial Basis Functions (RBF) [45], and wavelet functions. The performance of SVR largely depends on the selection of the kernel. One of the interesting features of the kernel-based approach is its ability to use feature mapping with infinite dimensions. For this purpose, a class of functions known as radial basis functions (RBF) can be used [46] as follows:

In the above equation, \({\varvec{x}}\) and \({{\varvec{x}}}^{^{\prime}}\in {\mathbb{R}}^{N}\), where \({\mathbb{R}}^{N}\) represents input space and \(\sigma\) is the RBF kernel parameter. After mapping the data to an infinite-dimensional space, the SVR tries to fit a hyperplane in the feature space to the training data. So, our prediction function is mapped as Eq. (4):where \(y\in {\mathbb{R}}\) and we seek to estimate (4) based on independent uniformly distributed data\(\left({{\varvec{x}}\boldsymbol{^{\prime}}}_{1}.{y}_{1}\right)\),…, \(\left({{\varvec{x}}\boldsymbol{^{\prime}}}_{n}.{y}_{n}\right)\) by finding function \(y({\varvec{x}})\) as sum of weighted kernels with a small error. The terms \(\alpha_{k}^{ + }\) and \(\alpha_{k}^{ - }\) are the corresponding Lagrange multipliers for the SVR model and \(\mathrm{b}\) is a penalty parameter used to suppress the noise and obtain a soft margin to create a more appropriate and accurate model.

In our work, input \(x\) and its corresponding target value \(y\) explicitly include the horizontal and vertical coordinates of inner and outer plexiform layer data in the OCT images. Although SVR with RBF kernel is excellent for layer segmentation in noisy images, it performed poorly in OCT slabs with disorganized layers, as demonstrated in Fig. 5d.

Zhang et al., showed that SVR with wavelet kernel has an accurate and acceptable performance against irregularities and fluctuations [47]. The wavelet kernel is a kind of multidimensional wavelet function that can estimate arbitrary nonlinear functions. So kernel function in Eq. (4) should be replaced with Eq. (6) as below:With the newly used kernel and fixing the epsilon, penalty parameters and the appropriate choice of \(\upsigma\), which is the degree of sensitivity of the kernel to small oscillations, significant results were obtained in dealing with small and large fluctuations in the IPL and OPL layers. To validate the automated algorithm's agreement with manual segmentation of IPL and OPL and also SI measurement in these layers, 50 OCT images with varying levels of retinal layer regularity were chosen (from relatively regular and organized layers in OCT slabs to highly disorganized layers in patients with diabetic retinopathy and diabetic macular edema). Two experts (EKP and HRE) manually segmented IPL and OPL using ImageJ's freehand tool and SI manually computed in each layer of the related OCT slab. The gold standard (GS) for segmentation of the layers and SI calculation was considered by averaging the results of two experts. Additionally, IPL and OPL layers were segmented using the SVR method with a wavelet kernel, and SI was calculated automatically for each layer in all OCT slabs. We utilized the Bland–Altman plot to determine agreement between two experts' manual segmentation and also between the automated method and the manual segmentation.

The results of applying the SVR with wavelet kernel and comparison of segmentation with two experts are shown in Fig. 6.

To analysis the IPL and OPL quantitatively, we introduced smoothness index (SI) that can be used to evaluate abnormalities in these layers under pathologic conditions like AMD, Epiretinal Membrane (ERM), and Diabetic Macular Edema (DME) [17‐19, 27, 29]. SI was used to evaluate the quantitative smoothness of IPL and OPL layers in macular OCT images and is defined as follows:where \({\mathrm{L}}_{\mathrm{E}}\) indicates the Euclidean distance between the beginning and the end of each layer in the final output image (the length of the straight line connecting the start and the end of IPL or OPL) and, \({\mathrm{L}}_{\mathrm{A}}\) is the actual length of the layer, obtained by calculating the sum of the Euclidean distance between all the pixels that make up the layer (Fig. 7).