Artificial neural networks in cardiology: analysis of graphic data

Aim. To consider application of convolutional neural networks for processing medical images in various fields of cardiology and cardiac surgery using publications from 2016 to 2019 as an example. Materials and methods. In the study, we used the following scientific databases: PubMed Central, ArXiv, ResearchGate. The cited publications were grouped by the area of interest (heart, aorta, carotid arteries). Results. The general principle of work of the technology under consideration was described, the results were shown, and the main areas of application of this technology in the studies under consideration were described. For most of the studies, sample sizes were given. The author’s view on the development of convolutional neural networks in medicine was presented and some limiting factors for their distribution were listed. Conclusion. A brief overview shows possible areas of application of convolutional neural networks in the fields of cardiology and cardiac surgery. Without denying the existing problems, this type of artificial neural networks may help many doctors and researchers in the future. Conflict interest. The authors declare the of obvious or potential conflicts of interest related the publication of this article.


INTRODUCTION
When examining a patient with cardiovascular diseases, a physician receives textual and numerical information (for example, medical history and blood test results), as well as graphic data (the results of computed tomography (CT), magnetic resonance imaging (MRI), echocardiography, scintigraphy, and X-ray), which require long-term analysis and assessment [1,2]. It takes a highly qualified expert up to 20 minutes to analyze MRI scans of a patient at two time points of the cardiac cycle -the end-diastole and end-systole [3]. It is a tedious and time-consuming process, that could lead to a diagnostic error [4]. However, in addition to the qualitative description, there is another important aspect of the quantitative assessment of images -linear and volumetric measurements for diagnosis, prognosis, treatment monitoring, and research purposes.
With the development of deep learning methods, such as neural networks, which have been used for image segmentation [5], object detection [6], and clinical decision support systems [7,8], and with their increased availability [9,10], it became possible to apply these methods in medical imaging [11][12][13]. In general, neural networks differ significantly from algorithmic approaches, which has been the main reason for their widespread use and implementation in the field of medicine. They have the ability to independently establish a relationship between input and output values via unsupervised training, which results in successful extraction of implicit or multifactorial relationships from data and better image interpretation [14].
Moreover, growth in computing performance, primarily due to graphics processing unit (GPU) computing [15], and availability of open-source neural networks make them accessible to many researchers [16]. Taking into account these factors, as well as the existence of large databases (for example, Ima-geNet [17], Cardiac CTA [18], ACDC [19]), the task of developing tools for reducing the contribution of the "human" factor to the analysis of medical images Бюллетень сибирской медицины. 2021; 20 (4): [193][194][195][196][197][198][199][200][201][202][203][204] remains relevant. In the period from 2008 to 2018, the number of publications dedicated to the machine learning approaches in medical image analysis increased by 8 times [20]. This paper presents several previous publications on the use of neural networks for medical image processing in various fields of cardiology and cardiac surgery in the period from 2016 to 2019.

THE CONCEPT OF A CONVOLUTIONAL NEURAL NETWORK
The history of neural networks began with the primitive feed-forward artificial neural networks (ANN) (usually known as the perceptron [21]), which were the first and simplest types of ANN. Further development of architectures led to the formation of deep learning ANN, which are characterized by complex topology and larger number of interconnected neurons, compared with perceptrons. These ANN imitate human cognition, making an association based on previous experience with the help of training, during which the probability of accurate object classification increases [22][23][24]. To date, convolution neural network (CNN) is considered to be the most effective ANN for image recognition. The main feature of this architecture is a convolutional layer. This layer (or set of layers) processes the input image (ANN extracts desired features) and then passes it to subsequent processing, similar to other ANN [25].
Given that CNNs are a type of ANN, they exhibit two main features -a need for training and the ability to switch [1]. To train the ANN, it is necessary to present it with a large number of labelled training data, where experts pre-select the features -similar to training of humans [24]. Therefore, the most important factor affecting the CNN is the quality of input data, primarily accurate segmentation. Another important aspect at the stage of developing CNN architecture is the structure and volume of data: a small sample or insufficient heterogeneity will lead to a large percentage of errors as a result, i.e. to a decrease in the quality of object recognition [1].
The ability of CNN to switch implies the ability to work with similar data. It is possible to conduct pre-training on data from open sources, and then fine-tune it for the target task [26,27]. Both features make CNN a promising and accessible tool for medical image analysis, and a number of multidisciplinary teams have been conducting research in this field.

Heart
Segmentation and quantitative assessment of cardiac and myocardial parameters are important in cardiology for assessing the severity of the initial state of the disease (dilatation, hypertrophy, contractile disorders, anatomical changes, etc.) and monitoring the results of treatment (remodeling, changes in the size of chambers). Despite the achieved progress in this area, this task is still challenging due to wide subject-to-subject anatomical variation. The main research directions in this area are image segmentation and classification.
For example, L. Yu et al. (2016) used CNN for fetal left ventricular (LV) segmentation in echocardiographic sequences [28]. Fetal echocardiography is the primary modality for evaluating prenatal cardiac function due to its low cost, harmless nature, and quick acquisition. A quantitative analysis of fetal echocardiographic images provides important fetal cardiac function parameters for early diagnosis of heart diseases.
The author proposes a dynamic CNN, the training of which includes 2 steps: pre-training and fine-tuning. Pre-training was carried out using images, where the neural network divided each pixel into two categories: a pixel in the region of interest and out of it. The training set consisted of 200,000 samples that were chosen randomly in 10 manually delineated sequences. The validation set consisted of 8,000 samples. It is worth noting that only the first frame of each echocardiographic sequence was segmented manually, which simplified the work of the experts. Thus, the dynamic CNN was fine-tuned by deep tuning to adapt to the first frame and by shallow tuning to fix the latest frame, adapting to the individual features. As a result, the segmentation accuracy was 94.5%. Further work is aimed at carrying out a quantitative analysis of fetal LV functions based on the results obtained using the proposed segmentation method. An example of the results obtained is shown in Fig. 1a.
W. Xue et al. (2018) [29] proposed an architecture for a deep multitask relationship learning network (DMTRL) which incorporates CNN for cardiac image representation and two parallel recurrent neural networks (RNN) for temporal dynamic modeling of cardiac sequences. The proposed network quantifies three types of LV parameters (the cavity inside the myocardium, regional wall thicknesses, and a cardiac cycle phase). The authors collected MRI images of 145 individuals (average age was 58.9 years), with 20 frames for each cardiac cycle. Compared with the previous study [30], this ANN demonstrated higher prognosis accuracy with an absolute error of 1.7-10.3% for the studied LV parameters.
J.D. Dormer et al. [31] presented a CNN-based heart chamber segmentation method for 3D CT with 5 classes: left ventricle, right ventricle, left atrium, right atrium, and background. Chest CT images were acquired for 11 patients with the total number of slices ranging from 78 to 154 for each patient, providing a large amount of data. The images were processed into pixel patches of five classes, 2 500 patches from each class for each patient were chosen for CNN training and validation. The results were validated by calculating the overall accuracy of the classification for each segmented region, with the accuracy defined as the number of correctly labeled patches from the total number of patches for the testing dataset. As a result, the accuracy in segmentation of the heart and the overall accuracy were 85.6 ± 6.1% and 87.2 ± 3.3%, respectively. It is worth noting that 11 unique cases resulted in such high accuracy of the network, despite insufficient heterogeneity of data. Nevertheless, this approach seems appropriate only for rare diseases, especially using augmentation of the dataset size due to rotation and scaling without substantial changes [25].
L. Tan et al. (2018) [32] developed a fully automated algorithm for LV segmentation in cardiac MRI. The study utilized the data of 200 subjects with coronary artery disease and regional wall motion abnormalities and 1,140 subjects with a combination of normal and abnormal cardiac functions. The combined training data and the manually labeled data were split 85:15 by the subject for training and cross-validation, respectively (i.e. 26, 069 and 9, 860 unique images). The developed algorithm demonstrated the median Jaccard similarity coefficient of 0.77 ± 0.11. The result of the input data processing is shown in Fig. 1b. Contrary to [31], this work has a large sample of input images for both training and validation. Aorta Aorta segmentation can be used for reconstructing its geometry, such as 3D models for further numerical analysis and preoperative planning, as well as for detecting pathological changes. Neural networks in this area can be used for assessment and selection of appropriate prostheses for transcatheter aortic valve replacement (TAVR) procedures.
Attempting to solve the problem of aortic segmentation, D. Wang et al. (2018) [33] developed a novel method for CT-MR aortic aneurysm image segmentation. The standard approach to training the CNN incorporates CT and MR images separately. However, this approach is time-consuming and inefficient due computational cost of training the ANN. The novelty of the proposed CNN is fusion of the parts of the model that work with CT and MR images. Such network can undergo end-to-end (complete) training using unlabeled CT and MR images in a shorter time period, since training occurs on two types of input data simultaneously. Moreover, the fusion model allows for shared representation of CT and MR images showing similar parts of the aorta for all image modalities (Fig. 2a). Processing images, the CNN segments them into five different classes, namely, aortic wall, its lumen, thrombus, calcium deposits, and irrelevant parts as background. The validation accuracy of the fusion models is 98.5%, which is 1.2% more than that of other models.
Another study in this area was conducted by P.M. Graffy et al. (2019) [34] (Fig. 2b), who used the fully automated Mask R-CNN algorithm [35] for segmentation of aortic calcification. The segmentation algorithm was applied to 9,914 non-contrast CT scans of 9,032 asymptomatic adults, who were screened for conditions not related to cardiovascular diseases [36]. The images were used to estimate the abdominal volume and the number of calcifications and assess the Agatston score (showing the extent of coronary artery calcification) [37]. Statistical processing of the results showed that the mean values for the Agatston score were higher in men (924.2 ± 2,066.2 vs. 564.2 ± 1,484.2, p < 0.001), the calcium mass was 222.2 ± 526.0 mg vs. 144.5 ± 405.4 mg (p < 0.001), and the abdominal volume -699.4 ± 1,552.4 ml vs. 426.9 ± 1,115.5 ml (p < 0.001). The mean score increased with age by 10% per year for the entire cohort. Compared with women, men (age 40-60 years) had higher calcium scores (91.2% vs. 75.1%, p < 0.001) and significantly higher mean Agatston score (age 50-80, p < 0.001). Thus, in combination with statistical methods, CNN allows researchers to quickly obtain large amounts of quantitative data and measurements and process them with standard methods of medical statistics, making this combination a necessary tool for scientific research. The authors noted, that this study is only the first step towards creating a clinical tool for detecting calcifications in the aortic wall.

CORONARY ARTERIES
Diseases of the coronary arteries (CA) may result in critical conditions [38,39], primarily coronary artery disease, which is the most common cause of death worldwide. CNN has the potential to become a valuable tool for locating and determining the degree of pathological changes in the arteries, especially in multivessel diseases.
X-ray coronary angiography is a primary imaging technique for diagnosing coronary diseases, consisting of consecutive projection images. E. Nasr-Esfahani et al. (2016) [40] used convolutional ANN to find and extract CA in X-ray coronary angiography processing of the results showed that the mean values for the Agatston score were higher in men (924.2 ± 2,066.2 vs. 564.2 ± 1,484.2, p < 0.001), the calcium mass was 222.2 ± 526.0 mg vs. 144.5 ± 405.4 mg (p < 0.001), and the abdominal volume -699.4 ± 1,552.4 ml vs. 426.9 ± 1,115.5 ml (p < 0.001). The mean score increased with age by 10% per year for the entire cohort. Compared with women, men (age 40-60 years) had higher calcium scores (91.2% vs. 75.1%, p < 0.001) and significantly higher mean Agatston score (age 50-80, p < 0.001). Thus, in combination with statistical methods, CNN allows researchers to quickly obtain large amounts of quantitative data and measurements and process them with standard methods of medical statistics, making this combination a necessary tool for scientific research. The authors noted, that this study is only the first step towards creating a clinical tool for detecting calcifications in the aortic wall. images. However, low quality resolution and image noise complicated processing of such images. Initially, an input angiogram was preprocessed to enhance its contrast. Afterwards, the image was evaluated using patches of pixels, and the network determined the CA and background regions and extracted them. A set of 1,040,000 patches was used for deep CNN learning, which were obtained from 44 X-ray angiography images. The large sample allowed for high accuracy of CA and background region identification -93.5% and specificity of 97%. Fig. 3 shows the ANN work, compared with manually annotated images.
It is impossible to assess the coronary bed from images using one projection angle. A 3D model provides more information, so research in this area would be promising. Hence, J.M. Wolterink et al. (2019) [41] proposed a method for coronary artery centerline extraction in cardiac CT angiography using a CNNbased orientation classifier (Fig. 4). Starting from a single seed point placed manually or automatically anywhere in the coronary artery, a tracker follows the vessel centerline in two directions using the predictions of the CNN. Tracking is terminated when no direction can be identified with high certainty. The CNN is trained using manually annotated centerlines in test images.
Evaluation was performed using a test set consisting of 24 coronary CT angiography (CCTA) test images in which 96 centerlines were extracted. The extracted centerlines had an average overlap of 93.7% with manually annotated reference centerlines. This study was a part of the Rotterdam Coronary Artery Evaluation Framework, which allows for the evaluation of algorithms for coronary artery centerline extraction.
Intravascular optical coherence tomography (OCT) is an optical imaging modality commonly used in the assessment of coronary artery diseases during percutaneous coronary intervention (PCI). Y.L. Yong et al (2017) [42] proposed a linear-regression CNN to automatically perform vascular lumen segmentation in OCT. The study used the total of 64 pullbacks acquired from 28 patients (25% / 75% male / female, the average age 59.71 (± 9.61) years) using Dragonfly™ Duo Imaging Catheter. These pullbacks were randomly split into a training and a test set in the ratio of 7:3. Benchmarking the results against the gold standard for manual segmentation, the proposed algorithm demonstrated the average CA wall location accuracy of 22 microns and the Dice coefficient and Jaccard similarity coefficient of 0.985 and 0.970, respectively. The mean absolute error in luminal area estimation was 1.38%. Assessment of the fractional flow reserve (FFR) [45] is a special form of CNN application in the field of medical image processing. After detecting regions of blocked CA during angiography, an interventional cardiologist, following the guidelines, makes a decision on FFR application based on the percentage of lumen diameter reduction. However, such intervention may be excessive in some cases, since stenosis could be hemodynamically insignificant, despite the occlusion. Therefore, there is a tendency for defining FFR as a functional parameter of CA stenosis. The FFR is defined as a distal / proximal pressure ratio in the stenosed segment [46].
These parameters are measured during invasive coronary angiography. To reduce the number of invasive procedures, M. Zreik et al. (2018) [47] presented a method for automatic identification of patients with functionally significant coronary artery stenoses, employing deep learning analysis of the LV myocardium at rest using CCTA. The automatic analysis of the LV myocardium was used to assess the FFR in the study. The analysis incorporated manual annotations of the LV myocardium (Fig. 3) and traditionally measured FFR parameters (n = 156) with the values of 0.79 ± 0.10. The neural network was tasked with classifying patients into those with functionally significant stenosis (FFR < 0.78) and those without it (FFR > 0.78). Quantitative evaluation of the segmentation performed on the 20 test scans resulted in a Dice coefficient of 91.4 ± 2.1% [43,44]. However, the sensitivity was 0.60-0.80 with the corresponding specificity of 0.77-0.59, depending on the CNN settings. These results cannot be properly transferred into clinical practice as a classification model, although the network helps noninvasively estimate FFR. The subsequent work of this team following the same principle did not demonstrate a significant increase in the quality of classification despite changing the FFR cut-off values for functionally significant stenosis (FFR ≤ 0.8) and adjusting the input data (n = 136) [48].
L. Itu et al. [49] proposed an efficient method for determining FFR in 2016. Researches trained CNN directly on CT scans of the CAs, i.e. associated geometric features with hemodynamic significance. The input data were multislice computed tomography (MSCT) scans of 87 patients with 125 stenosed regions. The researchers manually annotated arteries, reconstructed 3D models of the coronary vascular bed, and performed numerical modeling of the fluid dynamics, assessing the pressure gradient.
The model was validated, and the diagnostic accuracy for the detection of functionally significant CAD was shown to be 75-85%. Then, 12,000 coronary geometries were generated to artificially increase the sample, so the model could assess the FFR. These data were used to train CNN. Thus, the researchers increased the training set from 87 to 12,000 objects (by 138 times). The results of the proposed neural network were 99.7% consistent with the results of the numerical analysis (R = 0.9998, p < 0.001). The trained CNN was tested using input clinical MSCT data (n = 87) with the resulting sensitivity of 81.6% and specificity of 83.9%. This result is largely due to the imperfection of the computational fluid dynamics algorithms. Nevertheless, this study seems to be the most promising, since FFR estimation is based primarily on the CA anatomy. Perhaps, by combining two studies described above, a synergistic effect could be achieved by incorporating both LV and CA geometry analysis to improve FFR estimation. It should be noted that the method proposed in [41], allowing for the reconstruction of 3D representation of CA, can be combined with the method in [49], which can lead to a breakthrough in the field of noninvasive FFR estimation.

Detection of surgical devices
For TAVR procedures, the task of detecting catheters remains urgent, as it could assist in determining the optimal implant positioning.
The authors (2017) in [50] attempted to detect guidewires using datasets comprised of X-ray images. Overall, 22 image sequences were used in the study. The testing task of the region proposal network was divided into three steps. At the first step, 256 region proposals of guide-wires were generated from a test image as input data. At the second step, all the proposals were classified by the region, the region proposal was considered as the target, if its corresponding score was larger than the threshold value. Finally, the detected proposals with the highest score were selected. Following this algorithm, a total of 5, 092 images were obtained from the 22 original X-ray images. Then, researches divided 22 sequences into two sets, one -for training (19 sequences) and the rest -for testing (3 sequences). The detection accuracy reached 89.2%. The detection results are shown in Fig. 5, a. In 2019, H. Yang et al. [51] developed a method for catheter segmentation in 3D ultrasound images (Fig. 5, b) intending to use it during minimally invasive interventions. Since it was a pilot study, four data sets from four porcine hearts were used as the study samples. The whole algorithm was divided into three steps: 1) extracting the discriminating features from each voxel; 2) classifying voxels into catheter-like and non-catheter voxels using the CNN; 3) employing cubic spline interpolation to identify the catheter in the images. The proposed method can localize the catheter with the mean error of 2.1 mm while scanning the images for 10 seconds. With the increase in the computing power and optimization of the algorithm, this method would be able to instantly process datasets.
In the same year, a team of researchers led by H. Lee [52] used CNN to track and detect a peripherally inserted central catheter (PICC) and its tip. A total of 600 DICOM images from 600 different patients containing the keyword "PICC" were used in the study. The authors randomly selected 50 cases from the entire cohort to be used as a validation dataset and 150 cases to be used as a test dataset. The remaining cases were utilized to train fully convolutional networks (FCN) [53]. The neural networks developed by this team obtained absolute distances from ground truth with the mean of 3.10 mm, a standard deviation of 2.03 mm, and a root mean square error of 3.71 mm per 150 test cases. Despite the fact that all the images have a different angle and image noise, the CNN is able to accurately segment PICC line, ECG sensors, various objects (any additional structures), and threads (Fig. 5, c).

PROMISING DIRECTIONS FOR CNN
ANN have proved useful in the field of graphic data analysis: from medical image segmentation to assessing and predicting the development of pathologies in experimental and pilot studies. With the increasing availability of high-performance systems, ANN could find application in the form of commercial products, but this would require solving a number of problems related to clinical data and interaction between ANN and infrastructures.
Besides training, ANN requires a sufficient number of heterogenic data. The existing databases of annotated MSCT, CT, and MR images used for CNN training are limited. They usually include 100-300 images [19], while training often requires 1,000-10,000 samples. Various methods of artificial data generation and augmentation used in the studies have their own limitations and, due to their nature, contribute to the accuracy of CNN performance. Hence, collecting, standardizing, and annotating medical data can result in a promising project, especially concerning the development of multicenter databases. Another feature is heterogeneity of the neural network architectures. Research teams have been using their own models of ANN, never combining them with other CNNs for a cumulative effect. The different networks mentioned in this review could have been fused into a comprehensive system of neural network analysis to increase accuracy of the results. However, in practice this rarely happens. Perhaps, it is due to incompatibility of input data or neural network architecture. The prospect of combining multidirectional approaches for medical image segmentation or disease prediction, i.e., development of an integrated approach to neural network performance, can significantly increase the sensitivity and specificity of the results.
Finally, despite the development of computing hardware and image processing, many performance problems persist. The main calculations, such as selection of weight coefficients at the CNN training stage, happen on the graphic core of high-performance video cards (GPU). Compared with training on the central processing unit, GPU accomplishes the task much faster. However, a real-time image analysis (detection, segmentation) is usually carried out on less sophisticated machinery. The development of cloud computing in combination with CNN optimization algorithms should significantly simplify practical implementation of such systems by reducing the requirements to the PC computing power.

CONCLUSION
Over the past few years, ANN has been incorporated in many areas of our lives -from entertainment (applications for photo processing in smartphones, etc.) to engineering design systems (for example, generative technologies). The wide spread of machine learning methods in everyday life occurred due to the growth of computing power, both in stationary and in wearable devices. Medical field is no exception -ANN have proven effective in a wide range of tasks, including graphic data processing. Despite the advances in this field, the development of ANN has been slow for a number of reasons, several of which are described above. In this brief review, the possible applications of CNN in the field of cardiology and cardiac surgery have been shown. Although there is room for improvements, the network could become a reliable assistant for practitioners and researchers in the future.
Data availability presents the main problem to the implementation of CNN in healthcare. Thus, the question remains open, whether it would be possible to collect enough annotated data to train the ANN. Recent studies have shown that the more data there is, the better the results will be. However, it is not known how the big data can be used.
The above-mentioned studies have demonstrated that deep learning methods assist in the research process, but due to the uniqueness of the used data, Fig. 5. Results of CNN performance in detecting medical devices: a -when detecting a catheter in the CA; the tip of the catheter is marked with a square [50]; b -successful segmentation (left) and its original image (right) [51]; c -when segmenting PICC (red), ECG sensors (green), and various objects (dark cyan) [52] а b с mm per 150 test cases. Despite the fact that all the images have a different angle and image noise, the CNN is able to accurately segment PICC line, ECG sensors, various objects (any additional structures), and threads (Fig. 5, c). C mm per 150 test cases. Despite the fact that all the images have a different angle and image noise, the CNN is able to accurately segment PICC line, ECG sensors, various objects (any additional structures), and threads (Fig. 5, c). C researchers must search for other complex methods that would allow efficient analysis of clinical data. It can be concluded that the prospects of ANN application in healthcare have no limitations.