Validation of the machine learning approach for 3D reconstruction of carotid artery from ultrasound imaging

It is important to investigate the state of the arteries in order to detect atherosclerotic plaques in the early stage and then treat them appropriately. One of the diagnostic techniques is the ultrasound (US) examination. In order to obtain a more detailed and comprehensive overview of the state of the patient’s carotid artery, 3D reconstruction using the available 2D cross-sections can be performed. In this paper, deep learning is used for the automatic segmentation of US images, and this data is then used to reconstruct the 3D model of the patient-specific carotid artery. The validation of the proposed approach is performed by comparing two relevant clinical parameters for accessing the severity of vessel stenosis – the plaque length and the percentage of stenosis. Good validation results demonstrate that this method is capable of accurately performing segmentation of the lumen of carotid artery from US images and thus it can be a useful tool for assessing the state of the arteries in clinical diagnostics.


I. INTRODUCTION
Atherosclerotic plaque deposition within the blood vessel wall leads to arterial stenosis and severe catastrophic events over time. It is thus important to investigate the state of the arteries in order to detect possible obstructions in the early stage and then treat them appropriately. One of the diagnostic techniques is the ultrasound (US) examination [1,2]. This technique is noninvasive and inexpensive and is most commonly applied to analyze the carotid arteries. Using this approach, the two-dimensional (2D) cross-sectional images are obtained. However, for a more detailed and more comprehensive overview of the state of the patient's carotid artery could be obtained if a three-dimensional (3D) model was available. This can be done by performing a 3D reconstruction using the available 2D cross-sections.
There are several ways to extract relevant information about the vessel's cross-sections from the US images. In last years, the most promising and widely used approach for this task is one that is based on deep learning, i. e. convolutional neural networks. Deep learning is one of the techniques that has been used for medical imaging analysis, including brain, lung, and breast imaging. In this paper, deep learning is used for the automatic segmentation of US images, and this data is then used to reconstruct the 3D model of the patient-specific carotid artery. Two important clinical parameters for accessing the severity of vessel stenosis were measured during US examination by the clinicians and these two parameters were also calculated after the 3D reconstruction. The proposed approach is validated by comparing the obtained values.
The paper is organized as follows. In Section II the applied methods are presented. The results obtained for one particular patient and the validation results are presented in Section III. Section IV includes a discussion about advantages, drawbacks and future improvements of the presented approach and concludes the paper.

A. Clinical dataset
In this paper, clinical data from US examination in the baseline time point for overall 108 patients was used. The data is collected in the Serbian Clinical Centre. All images were annotated by clinical experts. For each patient the common carotid artery (CCA), the internal carotid artery (ICA) and carotid bifurcation are captured in transversal The research presented in this study was part of the project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. 755320-2 -TAXINOMISIS. This article reflects only the author's view. The Commission is not responsible for any use that may be made of the information it contains. The research is also supported by the Ministry of Education, Science and Technological Development of the Republic of Serbia (project numbers III41007 and ON174028). projection and the CCA and the ICA are also captured in longitudinal projection. The average number of images per patient was 8,7 which leads to the estimated total number of 939 images. Afterwards, the whole dataset was anonymized respecting the data protection and safety.
Taking into consideration the proposed methodology and the low quality of the US images, image preprocessing is an important pre-step for many deep learning algorithms related to the image and instance segmentation, as well as objects detection tasks. First, it should be noted that not all images were used for the image processing, such as images showing blood velocities withing the artery which had lower zoom and were inappropriate for scaling. The next step of the preprocessing task is the image size standardization. This is performed by selecting a static 512x512 pixels window for both arterial models, left and right. Special attention is paid to the window coordinates in order to the whole arterial tree is visible in the region.
After the preprocessing step, all images are labeled thus creating a dataset with labeled regions for the lumen. In Fig.  1, the first row consists of the examples of the original images, while the second row presents labeled images for the lumen region.

B. Deep learning segmentation
In traditional supervised machine learning algorithms, the user-defined features for a pixel-wise image segmentation are employed. That is why feature selection and extraction are the important steps for these algorithms [3].
On the other side, one of the promises of deep learning is thus to replace generic imaging features (e.g. wavelets, spatial textures, statistical moments), used in traditional ultrasound CAD systems, with processing layers that are more complex as well as more specific to the data, leading to an optimal use of information and improved prediction power. Taking into consideration that ultrasound images of the carotid bifurcation have typically low image quality, incorporating significant noise, artifacts, shadowing, and reverberation, it is very hard to represent ultrasound data with standard imaging features. On the other hand, deep learning can extract new discriminative features combining both global and local imaging information of the plaque.
In this paper, the automatic carotid artery segmentation for lumen has been done using FCN-8s [4,5], SegNet [6], and U-Net [7] based deep convolutional networks. Beside the original versions of these architectures, we modified the U-Net [8] and SegNet networks from the aspect of depth in order to test their capabilities to recognize the regions of interest. Our variant of U-Net uses batch normalization followed by a ReLU activation after each convolutional layer which proves to work a lot better on our data than the original U-Net model [7]. All models are trained with a combination of binary cross-entropy (BCE) and soft dice coefficient as a loss function, which is expressed as: Although the used dataset consists of images with colored and grayscale images, all available images were used for a training phase simultaneously.
All available subfolders corresponding to the patients are randomly divided into training, validation and testing sets by a ratio of 8:1:1 at the carotid artery level (either for the left or for the right arterial model).

C. 3D reconstruction
The 3D reconstruction of patient-specific carotid artery is performed using a generalized model that was presented in literature [9,10] that is then adapted to the particular patient, by introducing information obtained from only several 2D transversal cuts. This is done this way because only a limited number of 2D transversal cuts were available within the dataset.
The transversal cut of the CCA is used to define the shape of cross-section of this part of the carotid artery. The external carotid artery (ECA) is assumed to have a circular cross-section, with the diameter that is equal to the extracted diameter of the CCA multiplied by 0.59. The longitudinal cut of the ICA is used to extract the centerline of the ICA, the length and the diameters of the vessel in this segment, while two transversal cuts of the ICA (at the bifurcation and at the most stenotic part) are used to define cross-sections at the corresponding positions along the centerline. According to data from literature [9,10], the length of the ECA was set to the length of the ICA multiplied by 0.5, and the length of the CCA was set to be equal to the extracted diameter of the CCA multiplied by 1.2.
The reconstruction process can be divided in several phases, according to the algorithm presented in [11]: Automatic segment extraction using deep learning approach Conversion of extracted segments to nonuniform Bspline curves Definition of cross-sections along the centerline, by projecting them onto the trihedron normal-binormal plane [11] Generation of the NURBS surface [11] Generation of the 3D mesh The result of the 3D reconstruction is a 3D mesh of hexahedral finite elements that can be further used for the CFD analysis of blood flow, for example in the in-house developed software PakF [12,13].

III. RESULTS
In this paper, the binary classification task for image segmentation was considered. Three common classification metrics are considered for quantitative evaluation, including precision (P), recall (R), and Dice coefficient. The results of the method proposed in this paper are compared with thresholding technique and FCN-8s model with VGG16 as a backbone classifier [4]. Similar to U-Net, FCN model Fig. 2. Reconstructed geometry of the carotid artery for a particular patient; above the geometry are the US images for the appropriate transversal cuts (first row) and the predicted lumen region (second row) proved to work better with batch normalization. However, FCN has almost twice as many trainable parameters than U-Net network, so the U-Net can be trained faster and is more memory efficient. On the other hand, U-Net architecture has almost twice as many trainable parameters than SegNet network, but the results provided by SegNet are a little worse. In the speed-accuracy trade-off, the U-Net model was used for the final lumen segmentation task. The results for the test set are shown in Table I.
All the tests were performed on GIGABYTE NVIDIA GeForce GTX 1080 Ti 11GB, GDDR5X, 352bit. Python V3.6.7 is used as the programming language and related code is implemented using Keras framework, which regards Tensorflow framework as the backend.
The geometry of the reconstructed patient-specific artery is shown for one unseen patient. The reconstructed geometry is shown in Fig. 2, together with the clinical US images and predicted images for the lumen using deep learning.
There are two parameters that were estimated by the clinical experts and that were marked on the clinical US images and those are the plaque length and the percentage of stenosis. The percentage of stenosis is measured as the percentage of area of the lumen, compared with the area of the wall in the particular cross-section. The plaque length is measured as the length of the artery where the diameter is significantly (more than 20%) decreased. In order to calculate the values of these quantities for the reconstructed model, the diameters of the reconstructed ICA branches were extracted on overall 20 cross-sections along the length of this branch. The extracted diameters of the ICA along the centerline for the particular patient considered in this study are shown in Fig. 3.
For the validation purposes, the mentioned quantities (plaque length and percentage of stenosis) are compared for the four considered cases. The regression and Bland Altman plots for the plaque length are shown in Fig. 4, while the same plots for the percentage of stenosis are shown in Fig. 5. The obtained correlation coefficient of R 2 =0.9852 for the plaque length shows that the model is capable of reconstructing the shape of the vessel with high accuracy. The correlation coefficient for the percentage of stenosis is smaller (R 2 =0.7982) since the percentage of stenosis in clinical conditions was measured with respect to the diameter of the wall in that particular cross section, while in the reconstructed model, the average reconstructed diameter was used for this calculation, since the wall is not yet reconstructed in this phase. However, the obtained good correlation of the results demonstrates the capabilities of the developed 3D reconstruction technique.

IV. DISCUSSION AND CONCLUSION
In [14], the authors presented a deep learning approach for automatic characterization of plaque composition in carotid ultrasound images. The proposed CNN model showed good accuracy for the identification of the different plaque constituents. The evaluation of the Intima-Media Thickness (IMT) of the CCA in B-mode ultrasound images is considered the most useful tool for the investigation of preclinical atherosclerosis [15]. The paper [15] proposed a fully automatic segmentation technique based on Machine Learning and Statistical Pattern Recognition to measure IMT from ultrasound common carotid artery images. More precisely, the concepts of Auto-Encoders (AE) and Deep Learning have been included in the classification strategy. The pixels are classified by means of artificial neural networks to identify the IMT boundaries. The attempts to segment medical ultrasound images have had limited success than the attempts to segment images from other medical imaging modalities. Evaluation of a carotid ultrasound requires segmentation of the vessel wall, lumen, and plaque of the carotid artery. Convolutional neural networks are state of the art in image segmentation yet there are no previous methods to solve this problem on carotid ultrasounds. In [16], the U-Net convolutional neural network for lumen segmentation from ultrasound images of the entire carotid system was used. The 3D reconstruction is performed by adapting the generalized model to the patient data from a limited number of 2D transversal cuts that were available within the clinical dataset. This can be observed as a drawback, but on the other hand, it could be observed as an advantage of the developed 3D reconstruction model, because it is capable to cover the most important characteristics of the patient-specific carotid artery, that were analyzed and denoted by the expert clinician, while the more common characteristics were taken to be more general. This way, the amount of necessary US images is smaller, lowering the overall examination time and also the extraction of data from US imaging can be performed faster. The main drawback of this approach is that it reconstructs only the lumen of the artery, while ignoring the vessel wall. In the future improvements of the presented approach, the vessel wall will also be extracted and reconstructed. This will enable a more detailed overview of the state of the artery and also enable more complex numerical simulations of the blood flow and changes in arterial wall.
The results obtained during validation and presented in this paper demonstrate that this method is capable of accurately performing segmentation of the lumen of carotid artery from US images and thus, this method can be a useful tool for assessing the state of the arteries in clinical dignostics.