Open Access
Add to BibSonomy
Abstract
We propose a multimodal endoscopic system based on white light (WL), multispectral (MS), and photometric stereo (PS) imaging for the examination of colorectal cancer (CRC). Recently, the enhancement of the diagnostic accuracy of CRC colonoscopy has been reported; however, tumor diagnosis for a variety of lesion types remains challenging using current endoscopy. In this study, we demonstrate that our developed system can simultaneously discriminate tumor distributions and provide three-dimensional (3D) morphological information about the colon surface using the WL, MS, and PS imaging modalities. The results demonstrate that the proposed system has considerable potential for CRC diagnosis.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
The progress of colorectal cancer (CRC) follows the multistep chronological model, wherein the growth of genetic mutations and epigenetic alterations form cancer precursors over time, such as polyps, finally metastasizing to lymph nodes and distant organs [1]. Early-stage cancer, which generally protrudes or is recessed from the lumen, progresses to more advanced stages, such as colon submucosa invasion, and metastasizes to other organs, ultimately leading to death [2]. The progression to high-grade intraepithelial neoplasia or invasive carcinoma is correlated with the tumor distribution, morphology, and histological findings [3]. Thus, the documentation and description of the tumor distribution and morphology using the Paris classification system are recommended. For example, on the basis of the shape, the lesions on the colon are differentiated as pedunculated (Ip), sessile (Is), and nonpolypoid (slightly elevated (IIa), flat (IIb), or depressed (IIc)) or as excavated/ulcerated lesions in the Paris classification system. Besides, sessile and flat polyps are described as granular or nongranular laterally spreading tumors [4]. Therefore, colonoscopy screening is a critical procedure for preventing the worsening of the situation by determining a suitable treatment plan for CRC, according to the distribution and morphology of the tumors on colon-tissue surfaces. CRC detection is currently being realized using white light (WL) endoscopic imaging, and the tumor distributions are identified by endoscopists who are trained to observe properties, such as the contrast and color changes in the region of interest (ROI). However, smaller and variously shaped tumors reduce the WL endoscopy discrimination rate [5].
To overcome the above limitation of WL endoscopy, fluorescence colonoscopy has been employed to recognize the CRC region on normal tissues for accurate resection. However, no significant enhancement in the CRC detection rate has been demonstrated, although advanced endoscopic imaging modalities such as narrow-band imaging, autofluorescence imaging, and chromoendoscopy, have been developed [6–8]. Among them, fluorescence spectroscopy showed considerable potential for recognizing mucosal abnormalities during endoscopy, with a sensitivity and specificity of 100% and 97%, respectively, because the disease-related alterations of metabolic molecules, proteins, and others in the tissue affect the emission-light profile [9]. This indicated that multispectral (MS) imaging could become a powerful imaging modality for cancer margin detection. MS imaging acquires a set of images at the broader range of the spectrum with a high spectral resolution and spatial information. Hence, this spectral analysis-based imaging technique is advantageous for identifying and classifying tumors and normal tissues through their spectral signatures, with a high detection rate [10]. Therefore, MS imaging has been suggested and demonstrated as an advanced imaging technique, which can be utilized for discriminating between normal tissues and tumors, by various previous studies [11–14].
Furthermore, a few techniques have been proposed to provide the three-dimensional (3D) morphological information of lesions during endoscopy, utilizing customized hardware. However, it is generally challenging to implement within the physical constraints of colonoscopy. Structured illumination-based endoscopy for 3D imaging [15] requires a complex illumination system for pattern construction. Time-of-flight based or polarization-based 3D endoscopy needs a dedicated camera, including a time-of-flight sensor or rotating polarizer filter [16,17]. On the contrary, photometric stereo (PS) endoscopy could be conveniently developed with the addition of four illumination channels in flexible conventional endoscopy [18]. For example, this system can be implemented with only a few additional light sources at the distal face of the probe and can calculate the surface normal, which represents the surface directions, as well as the reflectance map, called the albedo. By merging WL and 3D morphological imaging techniques, the detectability and classification accuracy of lesions can be increased owing to the unique surface shape of colorectal adenomas, known as the precursor of invasive adenocarcinoma [18,19]. Therefore, we employed the PS imaging technique to reconstruct the 3D morphology, although it does not allow measurement of quantitative information about the tumor regions such as the height of the target, with high accuracy. However, MS and PS imaging techniques have inherent advantages and limitations, when utilized in isolation, resulting in a low tumor diagnosis accuracy. On the other hand, certain multimodal imaging systems that simultaneously offer varied and synergetic information can be highly beneficial biomedical tools for tumor diagnosis [20–22]. In particular, the combination of fluorescence and optical coherence tomography (OCT) techniques showed the capability to differentiate early dysplasia, adenocarcinoma, and normal mucosa on the basis of the injection of a near-infrared (NIR) dye and topographical information only along the laser-scanning direction [23,24].
In this paper, we propose a novel multimodal endoscopic system for tumor diagnosis, which provides complementary information from each imaging technique. This endoscopic system combines three imaging modalities: (1) WL imaging, which is capable of examining the colon in real time and determining the ROI; (2) MS imaging, which enables the discrimination of different biological tissues by comparing the inherent autofluorescence spectral signatures; and (3) PS imaging, which qualitatively offers 3D morphological images of the lesions. The proposed system is evaluated using a polyp-mimicking phantom that includes fluorescent microspheres in the protruding region and nine sessile polyp phantoms with different sizes. Also, multimodal endoscopic imaging of four colon tissues including tumors excised from mice is performed. The results demonstrate that the proposed multimodal endoscopic system can detect the tumor distribution and provide 3D morphological information of the colon, suggesting that it has the potential to be an advanced tool for enhancing the CRC diagnosis accuracy.
2. Materials and methods
2.1 System and probe design
Three imaging modalities were integrated into a stand-alone endoscopic system in order to obtain WL-based color, spectral-classified, 3D morphological images of the ROI. A program customized using LabVIEW controlled the multimodal endoscopic imaging system. Figure 1(a) presents the overall schematic of the multimodal endoscopic system, which includes the real-time WL, autofluorescence MS, and PS imaging systems. For the MS imaging system, a highly sensitive CCD camera (Pco.edge 4.2, PCO), 365-nm light-emitting diode (LED), and customized filter wheel, which includes 15 optical narrow bandpass filters (BrightLine, Semrock), an empty hole, and a servo motor are used. The pixel size and resolution of the CCD camera are 6.5 μm × 6.5 μm and 2048 × 2048, respectively. The filter wheel includes the empty hole for acquiring a broadband image for the other imaging modalities. Ultraviolet (UV) light was delivered using four single optical fibers with outer diameters of 500 μm after passing through a plano-convex lens and collimator. For the real-time WL and PS imaging systems, an RGB camera (GS3-U3-60QS6C-C, FLIR systems) and four 1.6 mm × 1.6 mm × 1.6 mm white LEDs are used. The pixel size and resolution of the RGB camera are 4.54 μm × 4.54 μm and 2736 × 2192, respectively. All these LEDs can be simultaneously turned on for even illumination or one at a time to acquire four images at different illumination directions. All the white LEDs were located on the distal face of the endoscopic probe and connected with two electrical wires to control and supply power. These imaging modalities utilize a path, which includes a GRIN lens (GT-IFRL-100-020-50, GRINTECH); a fiber bundle (FIGH-40-920G, Fujikura); an apochromatic 10 × objective lens; a tube lens; and a beam splitter. The fiber bundle has approximately 40,000 pixels within the image circle, which has a diameter of 854 μm. The outer diameters of the fiber bundle and GRIN lens are 1 mm. The working distance of the GRIN lens is 20 mm.
Fig. 1 Multimodal endoscopic system: (a) system diagram, (b) photographs of the system, and (c) endoscopic probe.
The endoscopic probe is separated into three parts; the two types of holes located outside the probe are for white and UV illumination, whereas that at the center of the probe is for imaging. Four single fibers for UV-light illumination, the four white LEDs, the GRIN lens, and the fiber bundle were fixed to the probe using optical epoxy. The diameter and length of the probe are 14 and 10 mm, respectively. Figures 1(b) and 1(c) show the photographs of the developed multimodal endoscopic system and probe.
2.2 White light and multispectral imaging and analysis
For real-time WL imaging, we utilized the RGB camera and four white LEDs. Real-time WL imaging determines the ROI for the MS and PS imaging techniques. For MS imaging and analysis, the target was excited by UV illumination. The light emitted from the target was acquired by a highly sensitive CCD camera, after passing through the 15 optical-bandpass filters sequentially from 420 nm to 700 nm with a step size of 20 nm. The full-width-at-half-maximum of each optical bandpass filter was approximately 20 nm. The images were stacked and then classified using a spectral angle measure (SAM), using the following equation [25]:
where si is the spectral signature of the i-th pixel of the collected image, uj is the j-th predefined reference spectral signature, and L is the number of the wavelength. Using Eq. (1), each pixel was mapped to the predefined color of the corresponding reference spectral signature, when the value was calculated as the smallest angle compared to each reference spectral signature. Note that red and green colors are used to represent tumors and normal tissues, respectively, in this paper.2.3 Photometric stereo imaging
For PS imaging, each white LED was controlled to acquire four images at different illumination directions, and the RGB camera acquired the images. Using this set of images, the surface normal for each pixel can be calculated on the basis of conventional modeling, as per the following equation [26]:
where I is the intensity of the i-th and j-th pixels of the image, kd is the reflectance coefficient (albedo), N is the surface normal, and L is the vector in the direction of illumination. Note that we replaced kdN with G to simplify the equation. From Eq. (2), the surface normal and albedo can be deduced using the least-squares solution as follows:The light vector, L, for each LED is calculated using the chrome sphere by determining the specular reflection from the chrome sphere in an image as follows:
where n is the normal vector of the highlighted region on the sphere and r is the viewing direction in an image.In the classic PS solution based on the Lambertian reflectance model, reconstruction of the surface is challenging in practice owing to the non-Lambertian surfaces, cast-shadow, and nonuniform illumination of light [27]. Furthermore, the fiber bundle, which shows honeycomb patterns on the image, causes distortion, when calculating the surface normal, and certain optical lenses can cause radial distortion. Therefore, we applied several conventional image-processing algorithms to correct the set of images acquired at different illumination directions. The set of images was corrected using the intrinsic, extrinsic, and lens distortion parameters [28], after which the honeycomb patterns were eliminated by applying image histogram equalization and Gaussian spatial filtering, as described in a previous study [29]. In addition, a homomorphic filter was applied to normalize the brightness across an image to reduce the bias at a flat surface caused by nonuniform illumination [30]. Besides, the region of the shadow of an image due to the direction of illumination was determined by applying an algorithm, as described in previous studies [31,32]. The shadow of the original image was then corrected by replacing the shadow region with the average pixel intensities from the other images. Furthermore, a spatial-lowpass filter was applied to prevent image discontinuity, which can emerge from the boundary of the replaced region. These procedures were iterated for each image, obtained under different illumination directions. In order to reconstruct the 3D morphology of the target, the gradients in each pixel from the surface normal were integrated [33]. The 3D morphological image was overlaid with the skin color. The surface-normal image was represented using red, green, and blue for the x, y, and z components, respectively.
2.4 Preparation of the polyp-mimicking phantom
For evaluating the performance of the proposed system, a polyp-mimicking phantom was constructed using 3D printing and was painted with the skin color. The phantom has a protruding region. The diameters of the bottom and top surface of the protruding area are 3 and 2 mm, respectively. Moreover, the height is 1.5 mm. The top of the protruded region has a recessed shape like a crater with a depth of hundreds of micrometers, which can position fluorescent microspheres (about 15 μm in diameter) to mimic the chemical properties of CRC.
To evaluate the detectability for the height difference of the target using PS imaging, we performed additional phantom experiments. Polyp-mimicking phantoms with the shape of a sessile polyp were constructed by 3D printing and painted with the skin color. Note that a sessile or flat polyp significantly increases the misdiagnosis rate in the colonoscopic screening. Moreover, the misdiagnosis rate for the detection of polyps with a diameter of less than 5 mm was reported as ~35.4% using an endoscope [34]. The height of a flat and sessile polyp is typically less than half of the diameter of the polyp [35]. Thus, phantoms with sessile polyps having diameters of 0.5–4.5 mm with a size difference of 0.5 mm were constructed. Their heights were here determined according to the diameter of each polyp. For example, if the diameter of a polyp is 4 mm, the height is 2 mm.
2.5 Preparation of the colorectal cancer model
HT-29 cells (human colon cancer cell line) were cultured in an RPMI 1640 medium supplemented with 10% FBS and 1% penicillin/streptomycin. The cells were incubated at 37 °C in 5% CO2 atmosphere.
Six-week-old BALB/c nude nu/nu male mice (Central Lab. Animal Inc.) were located in the animal facilities of the Daegu Gyeongbuk Institute of Science and Technology (DGIST) for two weeks. All animal experiments were conducted as per the approved protocol (Approval Number: DGIST-IACUC-18090401-01) of the Institutional Animal Care and Use Committee (IACUC).
The orthotropic CRC model was prepared, as previously reported [36], with minor modifications. In brief, mice were made to fast overnight with an ample supply of drinking water; the next morning they were anesthetized by intraperitoneal injection using a ketamine and xylazine mixture. Fifty microliters of 4% acetic acid were then applied to the rectal area with a narrow disposable plastic gavage and allowed to disrupt the colorectal epithelium for 2 min, followed by washing with 1 mL of PBS. Three hours after rectal mucosal disruption, the HT-29 cells (2 × 106) were placed in 50 µL of ice-cold Matrigel (Corning Life Science, Tewksbury), mixed, and instilled on the rectal mucosal surface by inserting a 28-gauge needle up to 5 mm from the anal ring. To prevent the leakage of cells, the anal opening was blocked with tape, until the mice recovered from anesthesia. The mice were observed for four weeks, for tumor development. They were then sacrificed, and the separated colorectal area was opened by incision, attached to a Whatman filter paper, and fixed in a 4% PFA solution.
Immediately after multimodal endoscopic imaging, samples were embedded in paraffin. These samples, with a sectioned thickness of 4 μm, were stained with hematoxylin for 10 min, washed, and then stained with eosin for 2 min. After washing with water, every slide was gradually dehydrated in 50%, 70%, 90%, and 100% ethanol. The hematoxylin and eosin (H&E)-stained images were acquired using a light microscope (Leica ICC50 HD, Leica Microsystems, Wetzlar).
2.6 Statistical analysis
The spectral signatures of CRC and normal tissues were obtained from four colon samples for quantitative statistical analysis. A paired two-tailed t-test was performed for the quantitative analysis. Here the significance level was chosen as a p-value <0.05.
3. Results
3.1 Evaluation of the performance of a multimodal endoscopic system
Using the multimodal endoscopic system that we developed, images of a grid target and 1951 USAF resolution target were obtained to evaluate the performance of the system. Figure 2 shows the grid and resolution target images. The grid target image shows slight barrel distortion (Fig. 2(a)). The barrel distortion was corrected by image processing with the intrinsic, extrinsic, and lens distortion parameters [28]. Figure 2(a) shows the image with barrel distortion correction. The diameter of the field of view at the focal length was measured to be 9 mm (Fig. 2). Note that all images acquired in this study were corrected by the barrel distortion correction algorithm. The distinguishable minimum frequency in the corrected resolution target image is approximately 8 cycles/mm, as shown in Fig. 2(b).
Fig. 2 (a) Color image of the grid target (left) and image corrected for barrel distortion (right) using WL imaging. (b) Target resolution (8 cycles/mm) (left) and intensity profiles in the lateral and axial directions along dotted lines A’ (middle) and B’ (right).
The spectral transmission profiles of the optical bandpass filters provide the spectral resolution of the developed endoscopic system. The average spectral bandwidth is ~20 nm, as shown in Fig. 3(a). In addition, we measured the spectral profiles of the WL and UV-light sources to offer additional information about the system as shown in Fig. 3(b).
Fig. 3 (a) Transmission spectra of the optical bandpass filters, and (b) spectra of the WL and UV-light sources.
3.2 Evaluation of the proposed system using the polyp-mimicking phantom
For evaluating the multimodal endoscopic system, we performed multimodal imaging of the polyp-mimicking phantom with the endoscopic probe. Figure 4(a) shows the structure of the polyp-mimicking phantom, which was constructed using 3D printing and then painted with the skin color. The magnified image in Fig. 4(a) shows a fluorescent microsphere located on top of the protruding region. Initially, the ROI was determined using real-time WL imaging, as shown in Fig. 4(b). MS imaging of the polyp-mimicking phantom was performed to obtain the fluorescent image cube at different wavelengths in the range of 420–700 nm, as shown in Fig. 4(c). The normalized spectral signatures of the microspheres and the region painted with the skin color were acquired from the fluorescence image cube. As shown in Fig. 4(d), the normalized spectral signature of the fluorescent microspheres exhibited a peak spectral intensity at approximately 440 nm, whereas the peak spectral intensity of the region painted with the skin color was at approximately 580 nm. With these normalized spectral signatures as a reference, the spectral-classified image was estimated (Fig. 4(e)) using Eq. (1). In this spectral-classified image, the red-colored regions represent the fluorescent microspheres on top of the protruding region of the polyp-mimicking phantom, whereas the green-colored regions indicate the other regions. The protruding region, which included the fluorescent microspheres, was clearly identified in the polyp-mimicking phantom by the MS imaging system.
Fig. 4 White light and multispectral imaging of the polyp-mimicking phantom: (a) photograph of the polyp-mimicking phantom, (b) color image acquired by WL imaging, (c) fluorescence spectral image cube, (d) normalized spectral signatures of the fluorescent microspheres and skin-colored phantom, (e) spectral-classified image.
We then evaluated PS imaging in the ROI. Figures 5(a)–5(d) present a set of images acquired at different illumination directions. These images depict the cast-shadows, according to the direction of illumination. The surface normal and albedo map were estimated using the least-squares solution after several preprocessing steps for each image (Figs. 5(e) and 5(f)). The surface normal of the inclined-plane region of the protruding region reflects the direction of the side surface, while the flat region on the surface shows the uniform normal vector in the z-direction. In addition, the reflectance coefficient at the inclined plane of the protruding region in the albedo map has low values. The morphology of the protruding region was reconstructed, on the basis of the integration of the gradients of the surface normal, as shown in Fig. 5(g). With the 3D morphology, the tendency of the protruding region and that of the recessed region on top with a depth of hundreds of micrometers were distinctly identified.
Fig. 5 PS imaging of the polyp-mimicking phantom: (a–d) set of images acquired at different illumination directions, (e) calculated surface normal image, (f) albedo map, and (g) reconstructed the 3D morphological image.
For multimodal imaging with the endoscopic probe, the total time, including acquisition and reconstruction, was approximately 9.2 s. MS imaging requires an acquisition time of 500 ms for each wavelength, which includes an exposure time of 400 ms and motor movement time of 100 ms. Moreover, the reconstruction time of the spectral-classified image was approximately 1 s. For PS imaging, an acquisition time of 50 ms for each illumination direction, and a reconstruction time of approximately 1 s were required. Overall, the results using the polyp-mimicking phantom demonstrated that the multimodal endoscopic system is capable of performing WL, MS, and PS imaging. Further, they suggest that the inherent autofluorescence information, which is relevant to tissue molecules, at each pixel of the image on the surface can be analyzed and provide 3D morphological target information.
In addition, we evaluated the detectability of PS imaging for targets with different sizes. Figure 6(a) shows phantoms having sessile polyp with different diameters and heights. The smallest one, which is located at the left-uppermost position in the phantom, has a diameter of 0.5 mm and a height of 0.25 mm, whereas the largest one at the right-lowermost position has a diameter of 4.5 mm and a height of 2.25 mm. Figures 6(b)–6(j) show 3D reconstructed images of phantoms with polyps having different sizes. Here, it was found that sessile polyps with different diameters and heights were discernable using PS imaging and analysis. In particular, PS imaging and analysis allows the detection of the smallest sessile polyp with a diameter of 0.5 mm and a height of 0.25 mm. However, the absolute height in the 3D reconstructed images of the phantoms could not be clearly quantified, which is an inherent limitation of PS imaging. Although we could not measure the absolute height of the polyps, 3D morphological information about the polyps could be obtained using PS imaging. Therefore, these results suggest that PS endoscopic imaging has the potential to discriminate the type of polyp according to the Paris Classification System, which may be useful for the early detection of CRC with a size of ~0.5 mm.
Fig. 6 PS imaging of phantoms containing sessile polyp: (a) photograph of the phantom, and (b–j) 3D constructed images of phantoms containing sessile polyps.
3.3 Multimodal endoscopic imaging of colorectal tumors ex vivo
After the evaluation of our multimodal endoscopic system using the polyp-mimicking phantom, we performed multimodal endoscopic imaging of four colon tissues including tumors from mice, ex-vivo. Figure 7 shows the representative multimodal and H&E-stained images. We initially determined the ROI, which showed the suspected region with tumors on the colon wall, using the WL imaging technique in real time, as shown in Fig. 7(a). MS imaging and PS imaging of the ROI were then carried out in order to determine the distribution and shape of the tumors on the surface, sequentially. From the MS images, we acquired the mean and standard deviation of spectral signatures of the tumor and normal tissues from four colon samples, as shown in Fig. 7(b). The tumor spectral signature exhibited a higher intensity within the wavelength range of 440–520 nm than the spectral signature of normal tissues, whereas the tumor spectral signature exhibited a lower intensity within the wavelength range of 560 nm – 600 nm. Here, it was found that the spectra of tumors and normal tissues were significantly different at 500 and 580 nm (p-value = 0.044 at 500 nm, p-value = 0.0095 at 580 nm, p-values < 0.05). The tumor distributions on the colon were delineated from the normal tissues by the spectral classification algorithm, with the normalized spectral signatures as the reference, as shown in Fig. 7(c). Note that the red and green colors in the spectral-classified image represent the tumors and normal tissues, respectively. Subsequently, PS imaging of colon tissues in the ROI was performed to obtain the 3D morphological information. The surface normal and albedo map were estimated using a set of images acquired under different illumination directions, and several image-processing algorithms were applied, as shown in Figs. 7(d) and 7(e). In addition, the 3D morphology of the colon tissues including the polyp was reconstructed using the integration of the gradients of the surface normal, as shown in Fig. 7(f). The 3D morphology of the colon tissues showed a protruding shape similar to a polyp. Furthermore, the locations of two polyps, which may be tumor indicators, were aligned with the spectral-classified image shown in Fig. 7(c). Finally, histological examination was conducted by staining with H&E in the tissue sections corresponding to the regions marked with white square boxes, labeled 1 and 2, in Fig. 7(a). The H&E-stained image (1) on the left-hand side of Fig. 7(g) shows highly proliferative tumors, and the boundary between normal tissues and tumors aligned with the spectral-classified image is shown in Fig. 7(c). It also shows that the tumors in the H&E-stained image (1) are aligned with the polyp in the reconstructed 3D morphological image shown in Fig. 7(f). The H&E-stained image (2) on the right-hand side of Fig. 7(g) shows normal colon tissue structures such as the mucosa, submucosa, and muscularis externa. Thus, these results establish that the developed system has the potential to discern tumors and normal tissues and provide morphological information of the colon tissues in three dimensions.
Fig. 7 Multimodal endoscopic imaging of colon tissues including tumors, ex vivo: (a) color image, (b) mean spectra with the standard deviation for colorectal cancer and normal tissues (n = 4), (c) spectral-classified image, (d) calculated surface normal image and (e) albedo image, (f) reconstructed 3D morphological image, and (g) H&E-stained images of the regions labeled 1 and 2 in (a), showing highly proliferative tumors (left) and normal colon structures (right).
4. Discussion
In this paper, it was demonstrated that the biochemical (molecular) and 3D morphological information of colon tissues, including tumors on the surface, can be provided by the proposed multimodal endoscopic system based on WL, MS, and PS imaging techniques. The obtained results also suggest that our system can be applied to examine the gastrointestinal tract in-vivo using the developed endoscopic probe, which has a diameter and length of 14 and 10 mm, respectively. The three imaging techniques, in particular, are synergetic and complementary with respect to surface tumor characterization for the following reasons: (1) real-time WL imaging enables the determination of the ROI using the color and contrast changes in the ROI, which appears cancerous; (2) MS imaging precisely provides the tumor distributions through the comparison of the emission spectra of tumors and normal tissues at different/consecutive wavelengths, which may not be achievable with WL and PS imaging; and (3) PS imaging qualitatively provides 3D morphological images, increasing the detectability and classification accuracy of tumors. Thus, we could delineate the tumor distributions and obtain the 3D morphological information of clinically significant lesions, as determined by the different biochemical properties and unique patterns, using WL, MS, and PS imaging, simultaneously.
Recently, novel endoscopic systems that combine different imaging techniques, have been introduced. A multimodal endoscopic system based on OCT and fluorescence imaging techniques with a contrast agent was developed to detect the architectural morphology and vasculature of the rectum wall [37]. A miniature endoscope with a combination of photoacoustic, OCT, and ultrasound was presented [21]. Both systems did not provide information about the tumor distribution and 3D morphological information about the surface; they only provided cross-sectional morphological and biochemical information on a target. However, since tumor progression is highly correlated with the tumor distribution and the 3D morphological information about the colon surface, this information is crucial to classify the types of tumors and plan treatments [3]. The proposed multimodal endoscopic system can delineate tumor distributions and provide 3D morphological information about the surface, which are crucial for tumor treatment planning such as resection from lesions without the injection of contrast agents.
MS imaging with the endoscopic probe was utilized to detect tumor distributions on the surface of the colon using a customized filter wheel, as shown in Fig. 7(c). Generally, an electrically tunable filter such as liquid crystal tunable filter (LCTF), which selects the light wavelengths in the MS system, is preferred for the spatial mapping of the light emitted from tissue because it is more convenient and easy to use [38]. However, an LCTF is disadvantageous because of its low transmission rate, which is generally within the range of 30%, compared to traditional optical bandpass filters that can transmit light at approximately 90% [39]. Hence, the LCTF may not be suitable for use with an imaging fiber, which can considerably attenuate the incoming light. Thus, we utilized a filter wheel, which included optical bandpass filters and a servo motor to ensure image quality by preventing the further diminishment of the attenuated light by the imaging fiber.
In 3D reconstruction, cast-shadow removal algorithms have been applied as a preprocessing technique to reduce effects of the cast-shadows, which result from the different light illumination direction and the short distance between the target and the light sources. However, since the cast-shadows may not be completely removed from the set of images acquired by the PS imaging technique, the remaining cast-shadows may affect the calculations of the surface normal vector, which hinders the precise 3D reconstruction of the targets with the larger heights. However, the large-shaped polyps such as pedunculated polyps could be discriminated with a high accuracy using a WL-based conventional endoscope [34]. Therefore, the PS imaging in the multimodal endoscope system would be beneficial for the detection of colorectal tumors through the analysis of different and complementary information about the ROIs.
The total time for acquisition and reconstruction of the MS and PS images was measured to be 9.2 s. This time might not be optimal for multimodal endoscopic imaging at a fixed region in clinical applications, which have periodic motility. For the in vivo application of the multimodal endoscopic system, rapid acquisition and reconstruction are required. This can be realized by the replacement of certain components with a high and uniform intensity light source, a camera with higher sensitivity, and an objective lens that has a higher numerical aperture, to reduce the camera exposure time. In addition, the wavelengths of 420, 440, 660, 680, and 700 nm were not found to be crucial for discrimination between the tumor and normal tissues as shown in Fig. 7(b). Thus, the images at those wavelengths could be ignored, thereby reducing the additional acquisition time. Furthermore, the time for reconstruction of the spectral-classified and 3D morphological images, which involves image-processing techniques such as honeycomb reduction, homomorphic filtering, and shadow correction, can be reduced by adopting higher computing resources. We intend to address these issues in our future works for the in vivo application of the system.
5. Conclusions
In summary, we developed a multimodal endoscopic system based on WL, MS, and PS imaging techniques for the discrimination of cancerous lesions at an early stage. The system simultaneously offers synergetic and varied information with real-time color, spectral classified, and 3D morphological images of the target. The proposed system was evaluated with polyp-mimicking phantoms as well as excised colon tissues including tumors. The results showed that the multimodal endoscopic system allowed the detection of the tumor distribution with a high quantitate and provided 3D morphological information about the colorectal tumor regions, demonstrating that it could be utilized for CRC diagnosis. Thus, this system has the potential to become an advanced endoscopic system for classifying the tumor type and for CRC treatment planning. However, the PS imaging in the multimodal endoscopic system has limited capabilities regarding 3D morphological reconstruction of the tumor regions. This may be caused by non-Lambertian surfaces, cast-shadows, the non-uniform illumination of light, and a short focal distance. This can be improved with additional illumination sources at more and different directions or precise alignment of the illumination directions. Moreover, to validate the usefulness of the system in clinics, additional in vivo and clinical tests with a large number of samples need to be performed, which remain as further study.
Funding
National Research Foundation of Korea (NRF) (2014M3A9D7070668, 2017R1A2B401072); Ministry of Trade Industry & Energy (MOTIE, Korea), Ministry of Science & ICT (MSIT, Korea), and Ministry of Health & Welfare (MOHW, Korea) under Technology Development Program for AI-Bio-Robot-Medicine Convergence (20001533).
Acknowledgments
We sincerely thank Thiago Coutinho Cavalcanti who did 3D printing of polyp phantom samples and endoscopic probe.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
References
1. B. Vogelstein, E. R. Fearon, S. R. Hamilton, S. E. Kern, A. C. Preisinger, M. Leppert, Y. Nakamura, R. White, A. M. Smits, and J. L. Bos, “Genetic alterations during colorectal-tumor development,” N. Engl. J. Med. 319(9), 525–532 (1988). [CrossRef] [PubMed]
2. R. Sullivan, O. I. Alatise, B. O. Anderson, R. Audisio, P. Autier, A. Aggarwal, C. Balch, M. F. Brennan, A. Dare, A. D’Cruz, A. M. Eggermont, K. Fleming, S. M. Gueye, L. Hagander, C. A. Herrera, H. Holmer, A. M. Ilbawi, A. Jarnheimer, J. F. Ji, T. P. Kingham, J. Liberman, A. J. Leather, J. G. Meara, S. Mukhopadhyay, S. S. Murthy, S. Omar, G. P. Parham, C. S. Pramesh, R. Riviello, D. Rodin, L. Santini, S. V. Shrikhande, M. Shrime, R. Thomas, A. T. Tsunoda, C. van de Velde, U. Veronesi, D. K. Vijaykumar, D. Watters, S. Wang, Y. L. Wu, M. Zeiton, and A. Purushotham, “Global cancer surgery: delivering safe, affordable, and timely cancer surgery,” Lancet Oncol. 16(11), 1193–1224 (2015). [CrossRef] [PubMed]
3. M. J. O’Brien, S. J. Winawer, A. G. Zauber, L. S. Gottlieb, S. S. Sternberg, B. Diaz, G. R. Dickersin, S. Ewing, S. Geller, D. Kasimian, R. Komorowski, and A. Szporn, “The National Polyp Study. Patient and polyp characteristics associated with high-grade dysplasia in colorectal adenomas,” Gastroenterology 98(2), 371–379 (1990). [CrossRef] [PubMed]
4. B. Meier, K. Caca, A. Fischer, and A. Schmidt, “Endoscopic management of colorectal adenomas,” Ann. Gastroenterol. 30(6), 592–597 (2017). [PubMed]
5. J. Burggraaf, I. M. Kamerling, P. B. Gordon, L. Schrier, M. L. de Kam, A. J. Kales, R. Bendiksen, B. Indrevoll, R. M. Bjerke, S. A. Moestue, S. Yazdanfar, A. M. Langers, M. Swaerd-Nordmo, G. Torheim, M. V. Warren, H. Morreau, P. W. Voorneveld, T. Buckle, F. W. van Leeuwen, L. I. Ødegårdstuen, G. T. Dalsgaard, A. Healey, and J. C. Hardwick, “Detection of colorectal polyps in humans using an intravenously administered fluorescent peptide targeted against c-Met,” Nat. Med. 21(8), 955–961 (2015). [CrossRef] [PubMed]
6. J. E. East, N. Suzuki, M. Stavrinidis, T. Guenther, H. J. Thomas, and B. P. Saunders, “Narrow band imaging for colonoscopic surveillance in hereditary non-polyposis colorectal cancer,” Gut 57(1), 65–70 (2007). [CrossRef] [PubMed]
7. D. Ramsoekh, J. Haringsma, J. W. Poley, P. van Putten, H. van Dekken, E. W. Steyerberg, M. E. van Leerdam, and E. J. Kuipers, “A back-to-back comparison of white light video endoscopy with autofluorescence endoscopy for adenoma detection in high-risk subjects,” Gut 59(6), 785–793 (2010). [CrossRef] [PubMed]
8. S. J. Chung, D. Kim, J. H. Song, H. Y. Kang, G. E. Chung, J. Choi, Y. S. Kim, M. J. Park, and J. S. Kim, “Comparison of detection and miss rates of narrow band imaging, flexible spectral imaging chromoendoscopy and white light at screening colonoscopy: a randomised controlled back-to-back study,” Gut 63(5), 785–791 (2014). [CrossRef] [PubMed]
9. R. M. Cothren, R. Richards-Kortum, M. V. Sivak Jr., M. Fitzmaurice, R. P. Rava, G. A. Boyce, M. Doxtader, R. Blackman, T. B. Ivanc, G. B. Hayes, M. S. Feld, and R. E. Petras, “Gastrointestinal tissue diagnosis by laser-induced fluorescence spectroscopy at endoscopy,” Gastrointest. Endosc. 36(2), 105–111 (1990). [CrossRef] [PubMed]
10. B. Regeling, B. Thies, A. O. Gerstner, S. Westermann, N. A. Müller, J. Bendix, and W. Laffers, “Hyperspectral Imaging Using Flexible Endoscopy for Laryngeal Cancer Detection,” Sensors (Basel) 16(8), 1288 (2016). [CrossRef] [PubMed]
11. R. Martin, B. Thies, and A. O. Gerstner, “Hyperspectral hybrid method classification for detecting altered mucosa of the human larynx,” Int. J. Health Geogr. 11(1), 21 (2012). [CrossRef] [PubMed]
12. A. O. Gerstner, W. Laffers, F. Bootz, D. L. Farkas, R. Martin, J. Bendix, and B. Thies, “Hyperspectral imaging of mucosal surfaces in patients,” J. Biophotonics 5(3), 255–262 (2012). [CrossRef] [PubMed]
13. H. Akbari, K. Uto, Y. Kosugi, K. Kojima, and N. Tanaka, “Cancer detection using infrared hyperspectral imaging,” Cancer Sci. 102(4), 852–857 (2011). [CrossRef] [PubMed]
14. S. Kiyotoki, J. Nishikawa, T. Okamoto, K. Hamabe, M. Saito, A. Goto, Y. Fujita, Y. Hamamoto, Y. Takeuchi, S. Satori, and I. Sakaida, “New method for detection of gastric cancer by hyperspectral imaging: a pilot study,” J. Biomed. Opt. 18(2), 026010 (2013). [CrossRef] [PubMed]
15. C. Schmalz, F. Forster, A. Schick, and E. Angelopoulou, “An endoscopic 3D scanner based on structured light,” Med. Image Anal. 16(5), 1063–1072 (2012). [CrossRef] [PubMed]
16. J. Penne, K. Höller, M. Stürmer, T. Schrauder, A. Schneider, R. Engelbrecht, H. Feußner, B. Schmauss, and J. Hornegger, “Time-of-flight 3-D endoscopy,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2009), 467–474.
17. S. E. M. Herrera, A. Malti, O. Morel, and A. Bartoli, “Shape-from-Polarization in laparoscopy,” in Biomedical Imaging (ISBI),2013IEEE 10th International Symposium on, (IEEE, 2013), 1412–1415.
18. V. Parot, D. Lim, G. González, G. Traverso, N. S. Nishioka, B. J. Vakoc, and N. J. Durr, “Photometric stereo endoscopy,” J. Biomed. Opt. 18(7), 076017 (2013). [CrossRef] [PubMed]
19. N. J. Durr, G. González, and V. Parot, “3D imaging techniques for improved colonoscopy,” (Taylor & Francis, 2014).
20. J. Kim, A. Seo, J. Y. Kim, S. H. Choi, H. J. Yoon, E. Kim, and J. Y. Hwang, “A Multimodal Biomicroscopic System based on High-frequency Acoustic Radiation Force Impulse and Multispectral Imaging Techniques for Tumor Characterization Ex vivo,” Sci. Rep. 7(1), 17518 (2017). [CrossRef] [PubMed]
21. X. Dai, H. Yang, T. Shan, H. Xie, S. A. Berceli, and H. Jiang, “Miniature endoscope for multimodal imaging,” ACS Photonics 4(1), 174–180 (2017). [CrossRef]
22. A. Hoffman, H. Manner, J. W. Rey, and R. Kiesslich, “A guide to multimodal endoscopy imaging for gastrointestinal malignancy - an early indicator,” Nat. Rev. Gastroenterol. Hepatol. 14(7), 421–434 (2017). [CrossRef] [PubMed]
23. N. Iftimia, A. K. Iyer, D. X. Hammer, N. Lue, M. Mujat, M. Pitman, R. D. Ferguson, and M. Amiji, “Fluorescence-guided optical coherence tomography imaging for colon cancer screening: a preliminary mouse study,” Biomed. Opt. Express 3(1), 178–191 (2012). [CrossRef] [PubMed]
24. R. A. Wall and J. K. Barton, “Fluorescence-based surface magnifying chromoendoscopy and optical coherence tomography endoscope,” J. Biomed. Opt. 17(8), 086003 (2012). [CrossRef] [PubMed]
25. S. Kim, D. Cho, J. Kim, M. Kim, S. Youn, J. E. Jang, M. Je, D. H. Lee, B. Lee, D. L. Farkas, and J. Y. Hwang, “Smartphone-based multispectral imaging: system development and potential for mobile skin diagnosis,” Biomed. Opt. Express 7(12), 5294–5307 (2016). [CrossRef] [PubMed]
26. S. Barsky and M. Petrou, “The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows,” IEEE Trans. Pattern Anal. Mach. Intell. 25(10), 1239–1252 (2003). [CrossRef]
27. R. J. Woodham, “Photometric method for determining surface orientation from multiple images,” Opt. Eng. 19(1), 191139 (1980). [CrossRef]
28. J. Heikkila and O. Silven, “A four-step camera calibration procedure with implicit image correction,” in Computer Vision and Pattern Recognition,1997. Proceedings., 1997 IEEE Computer Society Conference on, (IEEE, 1997), 1106–1112. [CrossRef]
29. J.-H. Han, J. Lee, and J. U. Kang, “Pixelation effect removal from fiber bundle probe based optical coherence tomography imaging,” Opt. Express 18(7), 7427–7439 (2010). [CrossRef] [PubMed]
30. C. Prabhakar and P. Kumar, “An image based technique for enhancement of underwater images,” arXiv preprint arXiv:1212.0291 (2012).
31. B. Sirmacek and C. Unsalan, “Damaged building detection in aerial images using shadow Information,” in 2009 4th International Conference on Recent Advances in Space Technologies, 2009), 249–252. [CrossRef]
32. C. Unsalan and K. L. Boyer, “Linearized vegetation indices based on a formal statistical framework,” IEEE Trans. Geosci. Remote Sens. 42(7), 1575–1585 (2004). [CrossRef]
33. R. T. Frankot and R. Chellappa, “A method for enforcing integrability in shape from shading algorithms,” IEEE Trans. Pattern Anal. Mach. Intell. 10(4), 439–451 (1988). [CrossRef]
34. N. H. Kim, Y. S. Jung, W. S. Jeong, H.-J. Yang, S.-K. Park, K. Choi, and D. I. Park, “Miss rate of colorectal neoplastic polyps and risk factors for missed polyps in consecutive colonoscopies,” Intest. Res. 15(3), 411–418 (2017). [CrossRef] [PubMed]
35. S. Kudo, “Endoscopic mucosal resection of flat and depressed types of early colorectal cancer,” Endoscopy 25(7), 455–461 (1993). [CrossRef] [PubMed]
36. H. Kishimoto, M. Momiyama, R. Aki, H. Kimura, A. Suetsugu, M. Bouvet, T. Fujiwara, and R. M. Hoffman, “Development of a clinically-precise mouse model of rectal cancer,” PLoS One 8(11), e79453 (2013). [CrossRef] [PubMed]
37. Y. Li, J. Jing, J. Yu, B. Zhang, T. Huo, Q. Yang, and Z. Chen, “Multimodality endoscopic optical coherence tomography and fluorescence imaging technology for visualization of layered architecture and subsurface microvasculature,” Opt. Lett. 43(9), 2074–2077 (2018). [CrossRef] [PubMed]
38. R. Abdlaty, S. Sahli, J. Hayward, and Q. Fang, “Hyperspectral imaging: comparison of acousto-optic and liquid crystal tunable filters,” in Medical Imaging 2018: Physics of Medical Imaging, (International Society for Optics and Photonics, 2018), 105732P.
39. R. M. Levenson and J. R. Mansfield, “Multispectral imaging in biology and medicine: slices of life,” Cytometry A 69(8), 748–758 (2006). [CrossRef] [PubMed]
