Open Access
Add to BibSonomy
Abstract
The adoption of computerized tomography (CT) technology has significantly elevated the role of pulmonary CT imaging in diagnosing and treating pulmonary diseases. However, challenges persist due to the complex relationship between lesions within pulmonary tissue and the surrounding blood vessels. These challenges involve achieving precise three-dimensional reconstruction while maintaining accurate relative positioning of these elements. To effectively address this issue, this study employs a semi-automatic precise labeling process for the target region. This procedure ensures a high level of consistency in the relative positions of lesions and the surrounding blood vessels. Additionally, a morphological gradient interpolation algorithm, combined with Gaussian filtering, is applied to facilitate high-precision three-dimensional reconstruction of both lesions and blood vessels. Furthermore, this technique enables post-reconstruction slicing at any layer, facilitating intuitive exploration of the correlation between blood vessels and lesion layers. Moreover, the study utilizes physiological knowledge to simulate real-world blood vessel intersections, determining the range of blood vessel branch angles and achieving seamless continuity at internal blood vessel branch points. The experimental results achieved a satisfactory reconstruction with an average Hausdorff distance of 1.5 mm and an average Dice coefficient of 92%, obtained by comparing the reconstructed shape with the original shape,the approach also achieves a high level of accuracy in three-dimensional reconstruction and visualization. In conclusion, this study is a valuable source of technical support for the diagnosis and treatment of pulmonary diseases and holds promising potential for widespread adoption in clinical practice.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
According to a report by the World Health Organization (WHO), cancer is a leading cause of death globally, accounting for nearly 10 million deaths in 2020. Among them, lung cancer represented 1.8 million cases, establishing it as one of the primary contributors to global mortality [1]. The incidence of lung cancer is on the rise due to factors such as an aging population and environmental changes, emphasizing the pressing need to enhance early detection and precise treatment techniques.
Pulmonary CT imaging has emerged as the principal method for tracking pulmonary diseases in many countries and regions. This imaging modality allows for the identification of pulmonary lesions, resulting in improved detection rates and reduced mortality through the application of effective treatment strategies [2–6]. However, traditional CT imaging primarily comprises two-dimensional image slices, which, although providing anatomical information, are insufficient for accurately depicting the spatial relationships within pulmonary tissues. Consequently, the precise removal of pulmonary lesions based solely on two-dimensional images poses significant challenges. Adequate preoperative imaging is crucial to ensure the safety and thorough excision of target lesions.
With the growing popularity of artificial intelligence-assisted surgery, three-dimensional analysis and reconstruction has become a significant gear in the medical field. The purpose of 3D reconstruction is to improve the understanding in a 3D context so that the images can be analyzed in 3D and not 2D [7–10]. The utilization of this technology has proven effective in enhancing surgical planning, reducing surgery time, and facilitating the identification of spatial relationships between pulmonary structures, thereby allowing for the successful treatment and safe removal of target pulmonary lesions [11–13]. Additionally, pulmonary three-dimensional reconstruction significantly improves surgical safety, as it enables preoperative and intraoperative simulations, enhances educational value, and enables faster large-scale anatomical research through the use of 3D images [14,15]. By visualizing a three-dimensional model, patients can gain a better understanding of their condition and increase their confidence in the proposed treatment. With the increasing frequency of pulmonary segmentectomy surgeries, the acquisition of accurate spatial information regarding pulmonary vessels, lesions, and other tissues for individual patients is crucial for successful surgical removal.
Li et al. [16] have developed a 3D reconstruction system specifically designed to assist in thoracic surgery planning, showcasing its clinical accuracy, efficiency, and safety. The results of their study demonstrate the system's ability to predict thoracic anatomical structures accurately. Moreover, in a study involving 420 patients who underwent lung 64-channel contrast-enhanced CT examinations, Ma et al. [17] employed 3D reconstruction techniques to analyze branch characteristics and variations of pulmonary arteries. Their research provides medical professionals with comprehensive and detailed references for preoperative assessment and surgical planning.
On the other hand, since tumor shapes exhibit randomness and variability, 3D reconstruction of tumor structures and growth trends is also essential for surgery [18,19]. Kuo et al. [20] developed a Computer-Aided Diagnosis (CAD) system that provides volume doubling time and growth rate by 3D reconstruction and volume information of nodules, thereby enhancing diagnostic and treatment effectiveness. Hasni et al. [21] used 3D visualization reconstruction of CT images from COVID-19 patients to improve the understanding of the disease. Wang et al. [22] analyzed the correlation coefficients and error percentages between actual total lung volume (aTLV) and AI-assisted 3D reconstructed CT lung volume (AI-3DCTVol) in 75 lung CT examination subjects. They suggested that AI-3D reconstruction, as a convenient method, holds significant potential in lung transplantation.
Despite the numerous advantages and conveniences that lung 3D reconstruction offers in medical anatomy and surgical planning, several challenges still need to be addressed. For instance, the precise capture of small blood vessels remains a challenge in 3D reconstruction [23]. Despite its provision of valuable spatial information, issues related to precision and accuracy persist in practical applications due to the complex and diverse nature of lung structures. Furthermore, the complexity of lung CT images and the presence of noise interferences can make it challenging to determine the accurate positional relationships between lung vessels and lesions. Additionally, it is worth noting that approximately 30% of early lesions are found to be attached to blood vessels [24] (refer to Fig. 1). Lastly, lung 3D reconstruction necessitates a substantial amount of medical imaging data, primarily obtained from CT scans. However, the processing of these large datasets demands significant computational resources and time.
Fig. 1. Examples of Lesions (green circles) connected to surrounding blood vessels (red arrows): Lesion sizes are (a) 4.45 mm, (b) 6.35 mm, and (c) 11.12 mm.
Lung 3D reconstruction possesses immense potential to enhance the diagnosis and treatment of lung diseases. Nevertheless, ongoing research and development efforts are essential to address the current challenges and issues associated with this technology. These endeavors will ensure its continued effectiveness and safety in clinical practice.
In recent years, significant progress has been made in three-dimensional lung reconstruction [25,26]. Zhao et al. [27] introduced an algorithm for lung nodule segmentation and 3D reconstruction based on PET-CT image sequences. This approach employed 3D region growing and ant colony optimization, proving to be more efficient than traditional methods and enabling comprehensive and accurate lung nodule segmentation. Chen et al. [28] presented a fully automated reconstruction algorithm for three-dimensional lung reconstruction in lung segmentectomy surgery, which holds promise in surgical planning by assisting surgeons in rapidly and accurately identifying anatomical patterns. Liu et al. [29] introduced PID-ME, a motion estimation method based on projection information separation, enhancing the 3D reconstruction of rotating coronary angiography. Through the integration of the 2D-PID-MCR framework, Fast Simplified Distance-Driven Projection (FSDDP), and Projected Average Minimum Distance (PAMD) model, it effectively estimated projection information, improving accuracy and real-time performance. Dong et al. [30] proposed an improved super-voxel-based 3D region-growing method for the segmentation and reconstruction of ground-glass nodules using PET/CT multimodal data. Xu [31] designed a repair method based on a 3D surface reconstruction algorithm for boundary-adherent and non-boundary-adherent tumors, addressing the issue of poor repair results caused by insufficient lung parenchyma boundary information in 2D CT images.
With the rise of deep learning, recent works have emerged in three-dimensional reconstruction based on deep learning [32–36]. Huang et al. [37] introduced a high-quality 3D visualization method for lung vessels based on low-cost segmentation and fast reconstruction. They initially extracted lung vessel features from lung CT using self-supervised learning and then employed self-supervised transfer learning to segment sparse lung vessels, followed by 3D reconstruction based on the segmentation results. However, the performance of self-supervised and transfer learning methods is often influenced by the model's generalization ability, potentially limiting their applicability across diverse clinical scenarios. Hong et al. [38] proposed a GAN-LSTM-3D method for three-dimensional (3D) reconstruction of lung cancer tumors from 2D CT images, achieving precise and high-resolution visualization of lung cancer tumor structures. Nevertheless, this approach requires significant computational resources and time for the 3D reconstruction of lung cancer tumors, potentially limiting its speed and efficiency in practical applications. Gu and Gao [39] also introduced a method that combines generative adversarial networks (GAN) and long short-term memory networks (LSTM) to generate models of lung tumors by segmenting and extracting features from CT scan images. Deng et al. [40] proposed an improved U-net network model for lung 3D reconstruction, demonstrating its effectiveness in thoracic anatomy. Wang et al. [41] described DeepOrganNet, a deep neural network method capable of generating high-quality 3D/4D organ geometry models in real-time from single 2D medical images, significantly improving reconstruction efficiency. However, most of these deep learning-based 3D reconstruction methods require substantial sample data, and curating large-scale datasets demands considerable computational resources and time [42]. Additionally, data quality and consistency can be problematic, as differences between devices and operators may impact reconstruction accuracy.
Currently, most work on lung 3D reconstruction focuses on reconstructing a single target, with very few addressing the spatial relationships between lesions and surrounding blood vessels. Chen et al. [43] developed a method for simultaneously segmenting lung nodules and blood vessels. They utilized line structure enhancement (LSE) and blob structure enhancement (BSE) filters for the initial selection of blood vessels and nodule candidate regions, followed by the front surfaces propagation (FSP) process for accurate segmentation of blood vessels and nodules. However, there is still a risk of missegmentation, which could lead to false positives or negatives, impacting diagnostic accuracy. The most closely related approach to our work is by Afshar et al [25], who proposed a method for three-dimensional reconstruction of tumor geometry from a series of 2D images. They initially performed tumor segmentation and then employed Marching Cubes with interpolation algorithms for 3D reconstruction, significantly improving the quality and accuracy of tumor shape reconstruction. However, these approaches are still limited to single targets and do not reflect the spatial positions of lesions in relation to surrounding tissues.
In this study, we began with actual CT images and achieved semi-automatic precise annotation of the target area. We then introduced a morphology-based interpolation algorithm, combined with Gaussian filtering, to smooth the lesions and blood vessels. Compared to deep learning-based methods, morphological methods typically have lower computational complexity, better interpretability, and often do not require extensive annotated data for training, expediting the reconstruction process. This step not only preserved their morphological characteristics but also ensured the accuracy of the reconstruction. After reconstruction, lesions and blood vessels can be segmented on any layer of the images, allowing for a more intuitive observation of the correlation between blood vessels and lesions across different layers. Additionally, considering that actual lung vessels may have branching, we applied a physiological vascular branching model to ensure continuity at the branches. Finally, experimental results demonstrate that this method can accurately reconstruct lung lesions and blood vessels, achieving realistic visual effects. Furthermore, it can be customized and optimized to meet specific medical research or surgical planning requirements based on individual application needs.
2. Materials and methods
First, begin by conducting semi-automatic, precise annotations of the lesions and blood vessels within the target area. This process involves utilizing third-order spline interpolation for annotating the edges of the vascular wall semi-automatically. The advantage of this method lies in the continuous differentiability of the curves generated by third-order spline interpolation, enabling easy derivative calculations along these curves. This capability proves immensely valuable when analyzing vascular shapes and features.
Moreover, it's worth noting that uniform distribution is not mandatory during the interpolation process. Instead, annotations of the vascular wall can be strategically placed at denser or sparser locations, depending on the specific requirements for capturing local shapes. Local adjustments are made whenever necessary to align the vascular wall's shape more accurately, all the while maintaining an overall sense of smoothness. This approach guarantees a high level of consistency in the relative positions of both the lesions and the surrounding blood vessels.
Following the completion of the semi-automatic precise annotation stage, the next step involves a comprehensive three-dimensional reconstruction. This phase primarily focuses on reconstructing the transition details observed at vascular bifurcations and relevant angles. To accomplish this, we employ a vascular bifurcation model rooted in principles inspired by biological evolution.
For a visual representation of the overall technical workflow, please refer to Fig. 2.
2.1 Biology-based vascular bifurcation model
In order to maintain blood circulation within the vascular system, an organism must supply the necessary energy. A portion of this energy is dedicated to nourishing the vascular walls, while the rest is employed to overcome resistance encountered during blood flow. It is clear that the total energy expenditure is intricately linked to the geometric shape of the vascular walls. Over the course of biological evolution, the geometric configuration of the vascular system has evolved to attain an optimal structure that minimizes energy consumption, a phenomenon well-documented in previous studies [43,44].
In the realm of CT cross-sectional imaging, achieving a perfect depiction of vascular bifurcations is not always feasible, as illustrated in Fig. 3. As a result, it becomes imperative to accurately reconstruct the three-dimensional structure, with particular emphasis on the relative sizes of vascular branches and the angles at which they bifurcate. Conforming to physiological principles, blood flows through the vasculature of the human body. To sustain this circulation, the ideal geometric shape of the human vascular system should exhibit an optimal bifurcation structure that minimizes total energy consumption.
Fig. 3. Consists of three consecutive images (a) to (c) from a CT sequence, illustrating the process of vascular bifurcation. However, it is challenging to make precise judgments regarding the exact location and angles of bifurcation.
2.1.1 Model assumptions
Rosen proposed a mathematical optimization model for vascular branching [45] based on the following assumptions:
- (1) At branching points, a primary blood vessel bifurcates into two smaller blood vessels. Additionally, within the vicinity of these branching points, there are three blood vessels that lie in the same plane and exhibit a symmetric axis, as depicted in Fig. 4.
- (2) When blood flow encounters resistance, it is treated as the movement of a viscous fluid within pipes that are, for practical purposes, rigid.
- (3) The energy provided by the blood to nourish the vessel walls increases as the inner surface area of the vessel and the volume occupied by the vessel wall grow. The volume of the vessel wall is determined by its thickness, which is roughly proportional to the vessel's radius.
Based on assumption (2), in accordance with the principles of fluid mechanics and Poiseuille's law, we can express the energy consumption of viscous fluid flowing through rigid pipes as follows:
where $\Delta p$ is the pressure difference between points A and C, µ is the viscosity coefficient of blood, and during the flow of blood, the energy expended by the body to overcome resistance is ${E_1} = q \cdot \Delta p$, substituting $\Delta p$ from Eq. (1):Assuming (3) is relatively complex, for a blood vessel with a radius of r and a length of l, the inner surface area of the vessel wall is $s = 2\pi rl \propto r$. The volume occupied by the vessel wall is $V = Al$, where A is the cross-sectional area of the vessel wall, and the wall thickness is denoted as d. Therefore, $\textrm{A} = \mathrm{\pi }[{{{({r + d} )}^2} - {r^2}} ]= \mathrm{\pi }({{d^2} + 2\textrm{rd}} )\propto {r^2}$. Consequently, the energy expended to provide nutrition to the blood vessel is:
where 1 ≤ α ≤ 2, and b is a proportionality coefficient.2.1.2 Model establishment
Based on the assumptions of the above model, the energy consumed when blood flows from point A in the main blood vessel to point B is represented as: $E = {E_1}^{\prime} + {E_2}^{\prime}$, where:
$E_1^{\prime}$ represents the energy expended to overcome resistance as blood flows through the vessel. $E_2^{\prime}$ represents the energy consumed to nourish the vessel walls during the blood flow process. $k $ is a constant representing the resistance generated as blood flows within the vessel, denoted as the resistance coefficient.
Therefore:
Furthermore, there are geometric relationships:
Substituting Eq. (5) into Eq. (4), and with ${q_1} = \frac{1}{2}q$
According to the principle of optimization, we aim to minimize E,
According to $\frac{{\partial E}}{{\partial r}} = 0,\frac{{\partial E}}{{\partial {r_1}}} = 0$:
Solving (7) yields:
From $\frac{{\partial E}}{{\partial \theta }} = 0$, and Eq. (8) yields:
Substituting Eq. (8) into Eq. (9) yields:
Therefore, Eqs. (8) and (10) represent the shape outcomes at vascular bifurcation points under the principle of minimum energy consumption in the body. With $1 \le \mathrm{\alpha } \le 2$, we can approximate the range of $\frac{r}{{{r_1}}}$ and θ as follows:
Based on the range of angles θ obtained above, when interpolating between two slices, for the case of vascular bifurcation, this can be used to determine the location of the vascular bifurcation, thereby achieving a smooth transition at the bifurcation site and ensuring connectivity within the blood vessels.
2.2 Three-dimensional reconstruction of lung tumors and blood vessels based on morphology-based slice automatic insertion algorithm
2.2.1 Image morphology
Dilation operation is the process of continuously merging background pixels that are in contact with the current region using a structuring element, resulting in the outward expansion of the region boundary (Fig. 5). It is commonly used to connect adjacent regions and can be mathematically represented as follows:
$A$ represents the pixel set of the current region. B represents the pixel set of the structural element. $A \oplus B$ denotes the result of the dilation operation, where $x,y$ represent specific pixels or pixel sets within the image.
In three-dimensional reconstruction, the dilation operation is commonly employed to expand the regions of blood vessels and tumors to ensure that these areas are fully connected and filled. Specifically, dilation can enlarge the boundaries of blood vessels and tumors, filling in potential small holes, thereby making them more continuous in the context of three-dimensional reconstruction.
Erosion operation involves the continuous reduction of elements within the current region that are in contact with the background region using a structuring element. The result of this operation is boundary contraction (Fig. 6). It is commonly used to eliminate small and insignificant objects in the image. Mathematically, it can be represented as follows:
In the three-dimensional reconstruction process, the erosion operation is used to reduce the size of blood vessel and tumor regions, aiming to remove unnecessary noise, smooth boundaries, and eliminate unwanted details. Erosion helps eliminate minor discontinuities along the boundaries of blood vessels and tumors, making them appear clearer and more regular in the context of three-dimensional reconstruction.
2.2.2 Morphology-based slice automatic insertion algorithm
Three-dimensional reconstruction of medical tomographic images is the process of constructing three-dimensional geometric models of tissues or organs based on a sequence of two-dimensional scan images obtained from medical imaging equipment, and then rendering and displaying them as “real” on a computer output device. With the target regions (polygons) obtained through semi-automatic labeling and a sequence of labeled blood vessels (represented by automatically improved edge polygons), reconstruction can be achieved by connecting (stacking) corresponding points in the slice sequence, as illustrated in Fig. 7.
Fig. 7. The process of CT slice insertion, where (a) represents CT1, (b) represents CT2, and (c) represents the inserted CT1mid.
Three-dimensional reconstruction relies heavily on addressing issues such as “gaps” caused by insufficient labeling of slices, which can significantly impact the accuracy of reconstruction and even lead to reconstruction failures. This project employs morphological principles to automatically add slices for tumor and vascular reconstruction using the following algorithm:
(1) Automatic Generation of Interpolated Slices
Suppose we need to insert a new CT slice between CT slice 1 and CT slice 2, with the centroid coordinates of CT1 and CT2 being (x01, y01) and (x02, y02), respectively.
Considering that most slices contain structures such as blood vessels or lesions resembling circular regions, often of smaller scale, a circular structural element of a 3 × 3 shape is chosen. This circular structural element aids in better handling of vessel shapes or circular lesion-like areas during dilation and erosion operations. The structural element is as depicted in Fig. 8.
- 1) Based on the trends in the areas (S1, S2, S3, S4) of CT slices 1, 2, 3, and 4, we determine whether the blood vessels need to become narrower or thicker:
- a. If the area trend suggests that blood vessels need to become narrower, we perform an erosion operation. The new CT slice to be inserted is named CT1mid, and we apply erosion to CT1, i.e., CT1mid = CT1_erode(erosion operation).
- b. If the area trend suggests that blood vessels need to become thicker, we perform a dilation operation. The new CT slice to be inserted is named CT1mid, and we apply dilation to CT1, i.e., CT1mid = CT1_dilate(dilation operation).
- 2) The centroid coordinates of the intermediate slice CT1mid, inserted between CT1 and CT2, are calculated as x0mid = (x01 + x02)/2 and y0mid = (y01 + y02)/2 to ensure that CT1mid is positioned in the middle between CT1 and CT2.
- 3) CT1mid, extracted at the centroid coordinates (x0mid, y0mid), is copied to that position.
The flowchart is shown in Fig. 9.
(2) Automatic Determination of Interpolation Position
To automatically determine the position of interpolation within a series of slices and achieve automatic interpolation, the following method is employed. the flowchart is shown in Fig. 10.
- 1) In the selected directory, calculate the distances between the centroids of each CT slice (xyDist), the x-axis offset (xyXaxis), and the y-axis offset (xyYaxis).
- 2) Sort xyDist, xyXaxis, and xyYaxis in descending order and select the CT slices with the largest offsets for interpolation.
- 3) Reconstruct the three-dimensional vascular map for all CT slices in the current directory.
- 4) If the reconstructed vascular structures exhibit gaps, repeat step (1).
3. Result
In this experiment, a set of CT sequences from actual patients was chosen. From these sequences, CT sequences with lesions adhering to adjacent blood vessels were selected. The target region was semi-automatically labeled, and then, using morphology-based slicing automation, the semi-automatically labeled two-dimensional image data was transformed into three-dimensional voxel data. Gaussian filtering was applied to smooth the data and eliminate unnecessary details while retaining important vascular and lesion information.
Using the three-dimensional voxel data, a three-dimensional reconstruction of the blood vessels and lesions was performed. Subsequently, internal vascular branch connections were smoothed based on the actual vascular intersections and branching patterns. Finally, the three-dimensional reconstructed data was visualized, allowing for the observation of the relationship between blood vessels and lesions at any desired plane.
3.1 Experimental environment
The server used in this experiment is a Lenovo ThinkSystem SR650, equipped with the following specifications: GPU: NVIDIA Tesla V100 32GB, Memory: 256GB DDR4 2666 MHz, CPU: 2 Intel Xeon Gold 5117 CPUs @ 2.00 GHz, and the operating system is 64-bit Windows Server 2016. Other software systems used include MATLAB R2019.
3.2 Experimental process
The specific experimental steps for this study are as follows:
- (1) Data Collection and Preprocessing: Initially, a suitable set of CT sequences from real patients was selected as the target reconstruction sequence and labeled as CT1, CT2, CT3, and so on, with ‘n’ representing the total number of CT slices used for reconstruction.
- (2) Semi-Automatic Labeling of the Target Region: Segmentation of lesion region was performed using a morphological region-growing approach. For labeling the vascular wall, professional doctors provided point annotations on the vascular wall, and semi-automatic labeling of the vascular wall was completed based on these annotated points using cubic spline interpolation.
- (3) Calculation of CT slice deviation: In the selected directory, the distances between the centroids of each CT slice were computed as xyDist = [5.2 mm, 4.8 mm, 6.1 mm, 5.5 mm, …], x-axis offsets as xyXaxis = [-2.1 mm, 1.5 mm, -3.2 mm, 0.8 mm, …], and y-axis offsets as xyYaxis = [0.7 mm, -0.9 mm, 1.2 mm, -0.5 mm, …]. These values were then sorted in descending order.
- (4) Interpolation between CT slices: Based on the sorting results, morphological interpolation was performed between CT slices with larger deviations following the size trend. For vascular branching situations, a biological vascular branching model was used to determine the branching angle, such as 40°, ensuring smooth and realistic branching.
- (5) Three-Dimensional Reconstruction: Independent reconstructions were conducted for blood vessels and lesions, followed by Gaussian filtering for smoothing. If reconstruction gaps appeared, steps (3) and (4) were repeated until no gaps were present in the reconstruction.
3.3 Experimental results
3.3.1 Vascular reconstruction results
To gain a clearer understanding of the transition process during vascular branching, a separate three-dimensional reconstruction was carried out specifically for vascular branching. The resulting reconstruction was rotated for observation, allowing for a detailed examination of the branching process within the vascular walls and the continuity of the vascular interior. The experimental results demonstrated that the reconstruction aligned with the vascular branching model, showing excellent connectivity within the blood vessels. The reconstruction using 16 layers of CT slices is illustrated in Fig. 11.
Fig. 11. The results of vascular reconstruction. From left to right, the results are progressively rotated by 45 degrees. In (a), the inner wall of the blood vessels and the smooth transition at the vascular branching points are clearly visible. In (c), the transparency of the blood vessel's inner wall can be observed.
3.3.2 Lesion and surrounding vascular reconstruction results
For lesions that do not intersect with blood vessels, three-dimensional reconstruction can be performed to demonstrate their relative spatial relationships (Fig. 12). In the left image, the lesion consists of 15 layers, and the blood vessels comprise 50 layers. The Y-axis spans approximately 80 pixels from 260 to 340. The right image shows the result after slicing at the 19th layer, making it more convenient to observe their spatial relationships for better subsequent processing.
In consideration of the intersection of lesions and blood vessels, a three-dimensional reconstruction was performed based on the semi-automatic annotation results of the lesions and blood vessels. In Fig. 13(a), there are 10 layers of lesions and 25 layers of blood vessels. The Y-axis ranges from 155 to 175, totaling approximately 20 pixels, with the lesion measuring 20 * 0.62 (resolution varies with different CT machines, and in this experiment, the CT slice is approximately 0.62 mm/pixel), resulting in 12.43 mm. Figure 13(b) shows the result of Fig. 13(a) rotated 90° in the horizontal direction, allowing for the observation of the positional relationship between the blood vessels and the lesions from different angles. Figure 13(c) represents the reconstructed image of Fig. 13(a) after cutting at the 8th layer and rotating the section, as shown in its entirety in Fig. 13.
Fig. 12. Shows the spatial positions of the lesions (in blue) and blood vessels (in red). The left image represents the reconstruction based on 50 layers of slices, while the right image shows the result after slicing at the 19th layer.
Fig. 13. Blood vessels (in red) passing through the lesion (in blue) and Figure (c) depicts the result of slicing and rotating Figure (a) along the horizontal direction at the 8th layer.
3.4 Experimental results validation
3.4.1 Vessel segmentation result validation
To validate the vessel segmentation results, we conducted experiments on the dataset provided in Ref. [24]. The results obtained through meticulous manual annotation by a medical professional were used as the Ground Truth. We compared our semi-automated third-order spline interpolation vessel annotation method with some classical and effective methods for pulmonary vessel segmentation [46–48]. The experimental results are shown in Fig. 14.
Fig. 14. (a) Original image, (b) ground truth, (c) Hessian matrix-based method, (d) local optimal thresholding method, (e) variational region growing method, (f) our method.
Through our approach, we first selected annotated points along the blood vessel wall around the lesion area by a professional medical expert. Subsequently, the vessel segmentation was carried out using a third-order spline interpolation method. This process ensures accurate preservation of the vessel's positional information, preventing any loss of vessels. This measure helps avoid the issue of missing vessels during subsequent reconstruction work. A performance comparison of segmentation methods is presented in Table 1.
Table 1. Performance Comparison of Segmentation
In this context, Sensitivity reflects the ability of various methods to identify positive instances, where in this experiment, positive instances correspond to blood vessels. F1-Score provides a comprehensive assessment of a method's accuracy and completeness. Jaccard Index, also known as IoU (Intersection over Union), measures the overlap between two labeled images, specifically evaluating the intersection over the union of the predicted and ground truth labels. Inference Time refers to the average inference time per set calculated for each algorithm by segmenting 5 sets of 50 images sized at 200 × 160.
Taking into account the aforementioned metrics, our semi-automatic cubic spline interpolation method ensures accurate identification of blood vessel regions, laying a solid experimental groundwork for subsequent reconstruction efforts.
3.4.2 Validation of three-dimensional reconstruction results
To calculate the accuracy of three-dimensional reconstruction, real three-dimensional information is required. However, obtaining this real three-dimensional information is not feasible. Therefore, for the validation of the experimental results in this study, 15 simulated realistic lesions and blood vessel 3D shapes were generated using three-dimensional reconstruction software. For each 3D model, a different number of cross-sections was used with the method described in this study for the reconstruction process. Subsequently, the classic Hausdorff distance was employed to measure the distance between the reconstructed shape and the original shape, along with other accuracy measurement standards to test the experimental results. The errors obtained from the 15 shapes were calculated to determine the average error, ultimately yielding the final results, as shown in Table 2 below. And as depicted in Fig. 15, with the continuous increase in the number of cross-sections, several accuracy indicators stabilize, reaching a saturation state, representing the optimal threshold of cross-section quantity.
Table 2. Validation of Three-Dimensional Reconstruction Experimental Results
The Hausdorff distance (in millimeters) is used to measure the maximum distance between the reconstructed and the real model structures, serving as an assessment of local inconsistencies. The Dice coefficient quantifies the overlap between the reconstructed vessels and lesions and the actual model structure. A Dice coefficient closer to 1 indicates a higher degree of overlap. Vessel curvature error (in millimeters) is employed to evaluate whether the reconstructed vascular structure preserves the curvature and shape of the actual vascular model.
On the other hand, considering the limited comparability of the Dice coefficient for small vessel structures, we performed reconstructions of smaller vessels using the same methodology, while ensuring an identical three-dimensional model height. We computed the Dice coefficients for these smaller vessels and compared them with those of the larger vessels, as depicted in Fig. 16.
The results from the example indicate that as the number of cross-sections increases, the accuracy of the three-dimensional reconstruction gradually improves, suggesting that using more cross-sections can capture the details and curvature of the simulated shape more accurately. On the other hand, with the continuous increase in the number of cross-sections, there exists an upper limit on the quantity of cross-sections. As the threshold approaches, the accuracy of the three-dimensional reconstruction tends to stabilize. However, this threshold is dependent on the volume of the reconstruction target.In the case of the 3D models in this study, reaching approximately 40 cross-sections represents this upper threshold limit.
Taking into account the above metrics, this experiment has achieved a higher level of accuracy and quality in the three-dimensional reconstruction of blood vessels and lesions. This enables researchers to better understand the relationship between lesions and surrounding blood vessels, providing valuable information for clinical diagnosis and treatment.
4. Discussion
The adoption of computerized tomography (CT) technology has significantly improved the role of pulmonary CT imaging in diagnosing and treating pulmonary diseases. Nevertheless, challenges persist due to the intricate relationship between pulmonary tissue lesions and surrounding blood vessels. Successfully addressing these challenges requires achieving precise three-dimensional reconstruction while maintaining accurate relative positioning of these elements. To tackle this effectively, our study employs a semi-automatic, precise labeling process for the target region, ensuring a high level of consistency in the relative positions of lesions and surrounding blood vessels. Additionally, we utilize a morphological gradient interpolation algorithm, combined with Gaussian filtering, to facilitate high-precision three-dimensional reconstruction of both lesions and blood vessels. Furthermore, this technique enables post-reconstruction slicing at any layer, facilitating an intuitive exploration of the correlation between blood vessels and lesion layers. We also leverage physiological knowledge to simulate real-world blood vessel intersections, determining the range of blood vessel branch angles and achieving seamless continuity at internal blood vessel branch points. The experimental results conclusively demonstrate the method's flexibility and precision in segmenting pulmonary lesions and blood vessels, even in complex scenarios such as lesion-blood vessel intersections. In summary, this study offers valuable technical support for the diagnosis and treatment of pulmonary diseases and holds promising potential for widespread adoption in clinical practice.
5. Conclusion
The research demonstrates that the semi-automatic precise labeling, morphological gradient interpolation algorithm, and smooth penetration at internal vascular branches employed in this paper can accurately segment lung lesions and blood vessels, achieving realistic three-dimensional reconstruction and visualization results. This breakthrough technology provides essential support for the diagnosis and treatment of lung diseases and opens up new possibilities for clinical applications. In the future, we anticipate that the findings of this research can be applied in the medical field to offer doctors more accurate lung CT images, aiding in earlier detection of lung diseases and providing additional information for treatment decisions. This has the potential to positively impact patients’ quality of life and health, offering improved medical care and treatment options for individuals with lung diseases in future medical practice.
Funding
Ministry of Science and Technology of the People's Republic of China (G2022016010L, G2023016002L).
Disclosures
The authors declare no conflicts of interest.
Data availability
The foundational data supporting the results presented in this paper is not currently accessible to the public, but it can be provided by the authors upon reasonable request.
Reference
1. J Ferlay, M Ervik, F Lam, et al., “Global cancer observatory: cancer today. Lyon: International Agency for Research on Cancer; 2020,” Lancet Global Health 11(2), e197–e206 (2023). [CrossRef]
2. Z Hong, Y Lu, Y Sheng, et al., “Comparison of three-dimensional reconstruction and CT-guided Hook-wire segmental resection for pulmonary nodules: a propensity score matching study,” World Journal of Surgical Oncology 21(1), 1–7 (2023). [CrossRef]
3. A Sadate, V Occean B, P Beregi J, et al., “Systematic review and meta-analysis on the impact of lung cancer screening by low-dose computed tomography,” Eur. J. Cancer 134, 107–114 (2020). [CrossRef]
4. E Zahnd W and M Eberth J, “Lung cancer screening utilization: a behavioral risk factor surveillance system analysis,” Am. J. Prev. Med. 57(2), 250–255 (2019). [CrossRef]
5. F Rong, R Shi, L Hu, et al., “Low-dose computed tomography for lung cancer screening in Anhui, China: A randomized controlled trial,” Front. Oncol. 12, 1059999 (2022). [CrossRef]
6. Y Chen, J Zhang, Q Chen, et al., “Three-dimensional printing technology for localised thoracoscopic segmental resection for lung cancer: a quasi-randomised clinical trial,” World J Surg Onc 18(1), 223 (2020). [CrossRef]
7. S Mukherjee, T De Silva, P Grisso, et al., “Retinal layer segmentation in optical coherence tomography (OCT) using a 3D deep-convolutional regression network for patients with age-related macular degeneration,” Biomed. Opt. Express 13(6), 3195–3210 (2022). [CrossRef]
8. S Mukherjee, T De Silva, G Jayakar, et al., “Retinal layer segmentation for age-related macular degeneration patients with 3D-UNet,” Medical Imaging 2022: Computer-Aided Diagnosis.12033, 1 (2022).
9. S Mukherjee, T De Silva, G Jayakar, et al., “Device-specific SD-OCT retinal layer segmentation using cycle-generative-adversarial-networks in patients with AMD,” Medical Imaging 2022: Computer-Aided Diagnosis. 12033, 68 (2022). [CrossRef]
10. Y He, A Carass, Y Liu, et al., “Longitudinal deep network for consistent oct layer segmentation,” Biomed. Opt. Express 14(5), 1874–1893 (2023). [CrossRef]
11. T Vervoorn M, M Wulfse, Mohamed Hoesein F A A, et al., “Application of three-dimensional computed tomography imaging and reconstructive techniques in lung surgery: A mini-review,” Front. Surg. 9, 1079857 (2022). [CrossRef]
12. J Le Moal, C Peillon, N Dacher J, et al., “Three-dimensional computed tomography reconstruction for operative planning in robotic segmentectomy: a pilot study,” J. Thorac. Dis. 10(1), 196–201 (2018). [CrossRef]
13. Y Ji, T Zhang, L Yang, et al., “The effectiveness of three-dimensional reconstruction in the localization of multiple nodules in lung specimens: a prospective cohort study,” Transl. Lung Cancer Res. 10(3), 1474–1483 (2021). [CrossRef]
14. S Nakazawa, T Nagashima, N Kawatani, et al., “Anatomy of the lung revisited by 3D-CT imaging,” Video-assisted thoracic surgery 8, 17 (2023). [CrossRef]
15. J Zhou and H Xin, “Emerging artificial intelligence methods for fighting lung cancer: A survey,” Clinical eHealth 5, 19–34 (2022). [CrossRef]
16. X Li, S Zhang, X Luo, et al., “Accuracy and efficiency of an artificial intelligence-based pulmonary broncho-vascular three-dimensional reconstruction system supporting thoracic surgery: Retrospective and prospective validation study,” EBioMedicine 87, 104422 (2023). [CrossRef]
17. X Ma, T Lu, D Qin, et al., “Analysis of pulmonary artery variation based on 3D reconstruction of CT angiography,” Front. Physiol. 14, 670 (2023). [CrossRef]
18. X Lei, X Yu, J Chi, et al., “Brain tumor segmentation in MR images using a sparse constrained level set algorithm,” Expert Systems With Applications 168, 114262 (2021). [CrossRef]
19. R Raja, S Kumar, S Rani, et al., “Lung segmentation and nodule detection in 3D medical images using convolution neural network,” Artificial Intelligence and Machine Learning in 2D/3D Medical Image Processing. CRC Press, 2020. 179–188.
20. F J Kuo C, J Barman, W Hsieh C, et al., “Fast fully automatic detection, classification and 3D reconstruction of pulmonary nodules in CT images by local image feature analysis,” Biomedical Signal Processing and Control 68, 102790 (2021). [CrossRef]
21. M Hasni, Z Farahat, A Abdeljelil, et al., “An efficient approach based on 3D reconstruction of CT scan to improve the management and monitoring of COVID-19 patients,” Heliyon 6(11), e05453 (2020). [CrossRef]
22. W Rui, S Yuhang, L Yang, et al., “A new method for evaluating lung volume: AI-3D reconstruction,” Front. Physiol. 14, 1 (2023). [CrossRef]
23. T Eguchi, T Sato, and K Shimizu, “Technical advances in segmentectomy for lung cancer: a minimally invasive strategy for deep, small, and impalpable tumors,” Cancers 13(13), 3137 (2021). [CrossRef]
24. D Rudyanto R, S Kerkstra, M Van Rikxoort E, et al., “Comparing algorithms for automated vessel segmentation in computed tomography scans of the lung: the VESSEL12 study,” Med. Image Anal. 18(7), 1217–1232 (2014). [CrossRef]
25. P Afshar, A Ahmadi, A Mohebi, et al., “A hierarchical stochastic modelling approach for reconstructing lung tumour geometry from 2D CT images,” Journal of Experimental & Theoretical Artificial Intelligence 30(6), 973–992 (2018). [CrossRef]
26. W Zhao and L Wang, “Research on 3D reconstruction algorithm of medical CT image based on parallel contour,” IEEE Sens. J. 20(20), 11828–11835 (2020). [CrossRef]
27. J Zhao, W Qiang, G Ji, et al., “3D reconstruction of pulmonary nodules in PET-CT image sequences based on a novel 3D region growing method combined with ACO,” International Journal of Bio-Inspired Computation 11(1), 54–59 (2018). [CrossRef]
28. X Chen, Z Wang, Q Qi, et al., “A fully automated noncontrast CT 3-D reconstruction algorithm enabled accurate anatomical demonstration for lung segmentectomy,” Thorac. Cancer 13(6), 795–803 (2022). [CrossRef]
29. X Liu, S Li, B Wang, et al., “Motion estimation based on projective information disentanglement for 3D reconstruction of rotational coronary angiography,” Comput. Biol. Med. 157, 106743 (2023). [CrossRef]
30. Y Dong, W Yang, J Wang, et al., “An improved supervoxel 3D region growing method based on PET/CT multimodal data for segmentation and reconstruction of GGNs,” Multimedia Tools and Applications 79(3-4), 2309–2338 (2020). [CrossRef]
31. Xu Yongxiao, “Research on Lung Tumor Segmentation Technology Based on Three-Dimensional Reconstruction,” Hebei University, 2020.
32. C Wang, M Hunt, L Zhang, et al., “3D localization of lung tumors on cone beam CT projections via a convolutional recurrent neural network,” Med. Phys. 47(3), 1161–1166 (2020). [CrossRef]
33. P Ardimento, L Aversano, L Bernardi M, et al., “Evo-GUNet3++: Using evolutionary algorithms to train UNet-based architectures for efficient 3D lung cancer detection,” Applied Soft Computing 110465, 110465 (2023). [CrossRef]
34. Z Wang, M Nakao, M Nakamura, et al., “Shape reconstruction for undetectable regions of abdominal organs based on a graph convolutional network,” Expert Systems with Applications 120593, 120593 (2023). [CrossRef]
35. F Chen, K Muhammad, and H Wang S, “Three-dimensional reconstruction of CT image features based on multi-threaded deep learning calculation,” Pattern Recognition Letters 136, 309–315 (2020). [CrossRef]
36. A Zheng, H Gao, L Zhang, et al., “A dual-domain deep learning-based reconstruction method for fully 3D sparse data helical CT,” Phys. Med. Biol. 65(24), 245030 (2020). [CrossRef]
37. Q Huang, L Zhang, L Liu, et al., “3D Visualization of Pulmonary Vessel Based on Low-cost Segmentation and Fast Reconstruction,” IEEE Access 11, 44866–44878 (2023). [CrossRef]
38. L Hong, H Modirrousta M, M Hossein Nasirpour, et al., “GAN-LSTM-3D: An efficient method for lung tumour 3D reconstruction enhanced by attention-based LSTM,” CAAI Trans on Intel Tech 1, 1 (2023). [CrossRef]
39. C Gu and H Gao, “Combining GAN and LSTM Models for 3D Reconstruction of Lung Tumors from CT Scans,” International Journal of Advanced Computer Science and Applications 14(5), 1 (2023). [CrossRef]
40. X Deng, Y Liu, and H Chen, “Three-dimensional image reconstruction based on improved U-net network for anatomy of pulmonary segmentectomy,” Math. Biosci. Eng 18(4), 3313–3322 (2021). [CrossRef]
41. Y Wang, Z Zhong, and J Hua, “DeepOrganNet: on-the-fly reconstruction and visualization of 3D/4D lung models from single-view projections by deep deformation network,” IEEE Trans. Visual. Comput. Graphics 26(1), 1 (2019). [CrossRef]
42. V Moravvej S, M J Maleki Kahaki, M Salimi Sartakhti, et al., “Efficient GAN-based method for extractive summarization,” Journal of Electrical and Computer Engineering Innovations (JECEI) 10(2), 287–298 (2022). [CrossRef]
43. B Chen, T Kitasaka, H Honma, et al., “Automatic segmentation of pulmonary blood vessels and nodules based on local intensity structure analysis and surface propagation in 3D chest CT images,” International journal of computer assisted radiology and surgery 7(3), 465–482 (2012). [CrossRef]
44. A Lauric, K Greim-Kuczewski, A Antonov, et al., “Proximal parent vessel tapering is associated with aneurysm at the middle cerebral artery bifurcation,” Neurosurg. 84(5), 1082–1089 (2019). [CrossRef]
45. R Rosen, Optimality principles in biology. Springer, 2013. https://doi.org/10.1007/978-1-4899-6419-9.
46. J Yang, S Ma, Q Sun, et al., “Improved Hessian multiscale enhancement filter,” Bio-Med. Mater. Eng. 24(6), 3267–3275 (2014). [CrossRef]
47. E Van Dongen and B van Ginneken, “Automatic segmentation of pulmonary vasculature in thoracic CT scans with local thresholding and airway wall removal,” 2010 IEEE international symposium on biomedical imaging: From nano to macro. IEEE, 2010. https://doi.org/ 10.1109/ISBI.2010.5490088. [CrossRef]
48. M Orkisz, H Hoyos M, P Romanello V, et al., “Segmentation of the pulmonary vascular trees in 3D CT images using variational region-growing,” IRBM 35(1), 11–19 (2014). [CrossRef]
