1. Introduction

Recently [1–3], holographic techniques [4] have been considered for real-time threedimensional (3D) sensing and identification of biological microorganisms. A single-exposure on-axis hologram formed on an image sensor is used to record the Fresnel field of the object illuminated by coherent light. The recorded Fresnel field of the biological micro-object is numerically reconstructed and image recognition techniques [5–8] can be applied. However, there are drawbacks in digital holography, such as the requirements for a coherent light source, stable recording environments, and the presence of speckle noise.

In this paper, we present experiments to illustrate the identification of biological microorganisms using integral imaging (II). Our objective is to show that II, with appropriate algorithms can be used to identify biological microorganisms. II [2, 9–18] is a conventional 3D display technique based on the multi-view directional information of a 3D scene. A microlenslet array generates a set of 2D elemental images on an imaging sensor. The scenes captured in elemental images have different perspectives according to the corresponding lenslets. The object in II can be illuminated by ambient or incoherent light. In contrast, holography requires coherent illumination to record the Fresnel field of the scene using interferometry.

In the computational reconstruction of II images [14–17], the volumetric information of 3D objects is numerically reconstructed by means of a ray projection method. Elemental images are projected through a virtual pinhole array at arbitrary longitudinal distances (depths) on a computer. Volumetric scenes can then be reconstructed to reproduce the original 3D information. Various techniques have been researched to improve the resolution of the computational reconstruction, including the moving array lenslet technique (MALT) [16–17].

There are significant benefits in developing automatic and real-time systems for biological microorganism recognition. Conventional bio-chemical inspection methods are generally labor intensive, time consuming, and require skilled personnel. Hence, there has been interest in the development of automatic and real-time systems for the recognition of microorganisms for various applications, such as monitoring harmful bacteria, biological weapon detection for security and defense, diagnosis of diseases, food safety investigation, and ecological monitoring. It is noted that most image-based efforts in this area use ordinary 2D images [19–21].

In this paper, we present two methods for the identification of biological microorganisms that use the 3D reconstruction volume from II. The first method uses multivariate statistical analysis, and the other is morphology-based recognition using the rigid graph matching (RGM) technique. Frameworks of the proposed approaches are presented in Fig. 1.

 

Fig. 1. Block diagrams for volumetric 3D recognition of biological microorganisms using (a) the multivariate statistical approach and (b) the morphology-based approach.

Download Full Size | PDF

For recognition using a multivariate statistical method, the multi-dimensional nonparametric statistical approach employing a multi-dimensional empirical cumulative density function (M-ECDF) [22] is introduced. Multivariate statistical analysis enables us to explore the joint relationship between multiple variables and the effect of each variable in the presence of the others [23], which can improve the discrimination performance between two data sets. In the case of a single variable distribution, there are many nonparametric statistical algorithms. However, these standard statistical methods cannot be applied to higher dimensional data sets. In previous research with 3D digital holography [3], we used parametric methods based on the Gaussian distribution of the data. The work presented in this paper is novel since we use non-parametric methods with non-Gaussian data. In our experiments, the classification performance using 3D volume imaging is compared to that using 2D data. The multivariate statistical approach is shown to be robust to changes in the morphology of the microorganisms.

For the morphology-based approach, feature extraction is performed by Gabor-based wavelet decomposition, followed by feature matching using the RGM technique which is used to identify unknown microorganisms. Decomposed components by Gabor-based wavelets represent the geometrical morphology of objects in the spatial frequency domain. The major advantage of 2D Gabor filtering is the ability to achieve the smallest possible spacebandwidth product. The Gaussian envelope in the Gabor kernels minimizes the uncertainty relations for the joint 2D spatial-frequency domain [24, 25]. RGM is a template matching technique between a reference object and an unknown input object [26]. This feature matching technique has been applied to search the reference morphology in unknown input scenes using 2D grids [1–3]. We extend the graph matching technique to utilize 3D volumetric images with 3D grids for the identification of unknown microorganisms.

The paper is organized as follows. In Section 2, we present a brief review of II recording and reconstruction. The design procedure of 3D recognition using a multivariate statistical method is described in Section 3. In Section 4, the decomposition process using Gabor-based wavelets and the RGM algorithm is described. Experimental results are illustrated in Section 5. Conclusions follow in Section 6.

2. Overview of integral imaging (II) recording and reconstruction

In II, a micro-lenslet array captures light rays emanating from 3D objects as shown in Fig. 2. The light rays that pass through each micro-lenslet are recorded on a 2D imaging sensor, such as a charge-coupled device (CCD) detector. Therefore, each micro-lenslet generates a 2D elemental image containing directional information of the 3D object. The captured elemental images have different perspectives of the 3D object according to the location of the microlenslet.

 

Fig. 2. A schematic setup of II for microorganism recording.

Download Full Size | PDF

The reconstruction process is the reverse of the recording process. In the optical display, 3D scenes are reconstructed by the intersection of discrete rays coming from the elemental images. The irradiance of the elemental images is propagated through the micro-lenslet array to form the 3D scene. The 3D display of II provides autostereoscopic 3D images with full and continuous parallax [9–11]. Computational reconstruction of II has been investigated to improve the quality of the optically reconstructed images, which can be degraded by optical display devices [15–17, 27]. Considering that rays from voxels on the 3D object surface contribute to pixels of elemental images during recording, volumetric images can be computationally reconstructed by an inverse mapping based on geometrical ray optics. In computational reconstruction, the elemental images are numerically projected through a virtual pinhole array to reproduce the original 3D information.

In this paper, II is used to obtain a 3D image of the microorganisms by a single exposure. Computational reconstruction of II generates 3D volumetric data, and reconstructed images have different focused regions depending on the longitudinal distances along the optical axis [15–17, 27]. By utilizing II sensing and reconstruction, we are able to sense micro-objects of interest, which can be located at arbitrary depth levels. The reconstructed volumetric images may present the 3D profile of the objects where the focused regions are decided by ray optics reproducing the structure of microorganisms in 3D space.

3. Microorganism recognition using the multivariate statistical method

In the following, we describe the design procedure for 3D recognition and identification of biological microorganisms using the II technique and the multivariate statistical method. There are advantages to using the II technique for the recognition of biological microorganisms. II can produce a number of depth images and multi-angle views of the scene. It enables us to obtain many variables that we require in order to recognize and identify the microorganisms. These variables can give us information about inter-correlations among the variables. Moreover, we can determine the contribution of each variable in the presence of the other variables. Hence, the discrimination performance of the system is enhanced by using the important information contained in these variables.

First, we reconstruct the volumetric 3D images of reference biological microorganisms that consist of p section images by the geometrical ray projection method. Let the multidimensional data set be described by X ^p, which is a combination of feature vectors [X ¹…X ^p]. We randomly select n test pixel points from a well-focused one section image of the reconstructed volumetric 3D image, where we choose enough samples to use in estimating a reference population distribution [28]. For classification of biological microorganisms based on multi-dimensional data sets, reference M-ECDF F_ref can be obtained by extracting n pixel values from each section image at the same test pixel points. It is also possible to analyze the variation of the data in the longitudinal direction at the fixed test pixel points. Given n ordered data points X^p (1), X^p (2), X^p (3), … X^p (n), the reference M-ECDF of the section images can be represented as follows:

where X^p is the randomly selected pixel value in the p^th section image and #{A} is the number of times the event A occurs. In order to obtain the statistical distribution of the criterion discriminant function of the reference data set for a statistical decision rule, we define the following criterion discriminant function for the null hypothesis:

where F′_ref is obtained by generating n′×p random sample data distributed according to the reference M-ECDF F_ref . We can obtain the statistical sampling distribution for the criterion discriminant functions by generating multiple F′_ref and substituting the resulting values in Eq. (2). The test statistics Λˆ enables us to convert a multi-dimensional data set into a one dimensional distribution and obtain the statistical sampling distribution for the null hypothesis. We then define the following discriminant function for comparing two multidimensional data sets:

where F_input is obtained by randomly extracting n′ pixel values from each section image of a unknown input microorganism at the same test pixel points. The values of Λ will lie between 0 and 1. We can obtain the statistical distribution for the discriminant functions by generating multiple F_input of the unknown input data set and substituting the resulting values into Eq. (3).

If the two multi-dimensional data sets, F_ref and F_input , are similar, the statistical distribution of Λ has sharp and peak points close to Λ=0.5. The statistical distribution of the criterion discriminant functions for the null hypothesis Λˆ has its highest value at Λˆ=0.5. Finally, we calculate the mean-square-distance (MSD) for comparing the actual multi-dimensional discriminant functions Λwith the criterion discriminant functions Λˆ:

where E{·} is expectation operator. The statistical decision for identification of biological microorganisms can be achieved with a hypothesis test on the criterion discriminant function. This statistical approach for 3D recognition is suitable for recognizing the microorganisms, such as bacteria and biological cells, which do not have well defined shapes or profiles. Therefore, it allows our 3D recognition system to be robust to variations in the shape of the microorganisms.

4. Morphology-based recognition using Gabor-based wavelets and the RGM technique

The elementary function of the Gabor-based wavelets has the form of a Gaussian envelope modulated by a complex sinusoidal function [24, 25]:

where x is a 2D discrete position vector, k _uv is a wave number vector, and σ is proportional to the standard deviation of the Gaussian envelope. k ^uv is defined as k _uv=k _0u[cosϕ_ν sinϕ_ν ]^t, where k _0u=k ₀/δ ^u-1, ϕ_ν =[(ν-1)/V]π, u=1, …, U, and ν=1, …, V. k _0u is the magnitude of the wave number vector, ϕ_ν is the azimuth angle of the wave number vector, k ₀ is the maximum carrier frequency of the Gabor kernels, δ is the spacing factor in the frequency domain, U and V are the total number of decompositions along the radial and tangential directions, respectively, and superscript t denotes transpose.

The Gabor-based wavelets perform band-pass filtering with spatial and orientation frequency bandwidths depending on the Gaussian envelope. The carrier frequency of the band-pass filter is determined by k _uv. By changing the magnitude and direction of the vector k _uv, we can scale and rotate the Gabor kernel to make self-similar forms, so that each Gabor kernel covers the frequency range selectively. The parameterization of 2D Gabor-based wavelets has been investigated in [24, 25]. It might be desirable to set the parameters of Gabor kernels for the complete representation of images. In this paper, the parameters are set at σ=π, k ₀=π/8, δ=√2, U=6, and V=12.

Let y_uv be the filtered output of the image o after it is convolved with the Gabor kernel g_uv:

where the operator * denotes 2D convolution, and o is the reconstructed image. The output y_uv is referred to as the Gabor coefficient. The rotation-invariant property can be achieved by adding up all of the Gabor coefficients along the tangential directions of the frequency domain. Therefore, we can define a rotation-invariant node vector at x _i, which is the location of pixel i, as

It is noted that v(x _i) is a U-dimensional complex vector.

We apply the RGM technique [1–3, 26] with 3D grids to the 3D reconstructed volume images. To use the RGM technique, the Gabor coefficients are defined on a graph, which is a set of nodes associated in the local area. The graphs are superimposed on a reference image and an unknown input image. The RGM technique performs template matching between the two sets of Gabor coefficients, measuring the geometrical shapes between the reference object and the unknown input sample.

The 2D graph matching technique is applied to 3D layers of images reconstructed at several depth levels. Two 3D grids are superimposed on volumetric images of the reference and input microorganisms; therefore, the reference morphology is defined on a 3D grid along a certain length of longitudinal distance. Similar morphology with the reference is searched using the 3D grid on volumetric images of unknown input microorganisms.

Let R and S be two identical and rigid 3D grids placed on the 3D reference image set Ω_r and the unknown 3D input image set Ω_s, respectively. The reference image set Ω_r and the input image set Ω_s are composed of images reconstructed at different depths:

where o_r and o_s are the reference and input images, respectively; $d_j^r$ and $d_j^s$ are the reconstruction depths for the reference and input images, respectively; and D is the number of depth levels. The location of the reference graph in each reference image o_r is predetermined by a translation vector p _r, and a counter clock-wise rotation angle θ_r . Position vectors of nodes in each reference image can be computed as

and

where ${\mathbf{x}}_k^o$ and ${\mathbf{x}}_c^o$ are the position vectors of the node k and the center of the grid without any translation and rotation, respectively; and K_grid is the number of nodes in the grid for one reconstructed image.

In our database, the reference features are stored in order to represent the morphology of the reference microorganism. Assuming the graph R covers a designated shape representing the geometrical characteristics of the reference microorganism, we search the similar local shape by translating and rotating the graph S in the unknown input image set, which has arbitrary depth levels. A similarity function between the graph R and S is defined as

where 〈·〉 stands for inner product; v _r[x _k(p _r, θ_r ; $d_j^r$ )] and v _s[x _k(p _s, θ_s ; $d_j^s$ )] are two complex rotation-invariant node vectors of the graph R in the reference image set and the graph S in the unknown input image set, respectively. The local area, which is covered by the graph S, is identified with the reference shape if the following condition is satisfied:

where α _Γ is a threshold for the similarity function; and θˆ_s is obtained by searching the best matching angle to maximize the similarity function at the position vector p _s as

In this section, we have utilized the Gabor coefficients using 3D grids superimposed on reconstructed images. By utilizing multiple layers of images for the reference and the input, we search the 3D reference shape residing at various depths levels to improve the system performance.

5. Experimental results

Two filamentous microorganisms, sphacelaria alga and tribonema aequale alga are captured by the microscopic II system. A schematic diagram of the experimental setup for II recording is shown in Fig. 2. The approximate size of sphacelaria alga and tribonema aequale alga is 50~100µm and 10~20µm, respectively. The size of one pixel in the CCD array is 7.4×7.4µm. The distance between the lens array and the reconstructed plane of the object is approximately 300µm, including the cover glass thickness. The magnification of the objective lens is approximately 20. The size of each elemental image is 180×180 pixels. Because objects are reconstructed with these elemental images, the resolution of the reconstructed object may not be sufficient to see a clear reconstructed image. However, the resolution of the reconstructed image can be improved using the MALT.

 

Fig. 3. Sections of elemental images for (a) sphacelaria alga and (b) tribonema aequale alga.

Download Full Size | PDF

 

Fig. 4. Reconstructed images at depth d=330 µm for (a) sphacelaria alga and (b) tribonema aequale alga.

Download Full Size | PDF

Figure 3 shows a section of elemental images of sphacelaria alga and tribonema aequale alga, respectively. Multiple-layers of microorganism images are reconstructed at different depth levels by the ray projection method. A total of 12 images are reconstructed at depths d=300, 306, …, 366 µm for each microorganism, where d is the distance between the lens array and the reconstructed plane of the object. In II, a micro-lenslet array captures light rays as shown in Fig. 2. The light sources that are diffused from the 3D objects pass through each micro-lenslet and are recorded on a 2D imaging sensor, such as a CCD detector. Each microlenslet generates a 2D elemental image containing directional information of the 3D object. In the microscope system, to get magnified elemental images of object, an objective lens is used between the lens array and the CCD. The minimum distance of the reconstructed object is the same as the thickness of the cover glass of the slide.

For preprocessing, the reconstructed images are cropped by 800×800 pixels and resized to 256×256 pixels. Irradiances of the reconstructed images are inversely normalized from -1 to 1. Hence, the dark foreground region has higher pixel values close to 1 and the bright foreground region has pixel values close to -1. Figure 4 shows two reconstructed images at a depth of 330µm after the preprocessing. In the next two subsections, we present the experimental results of the multivariate statistical method and the morphology-based approach.

5.1 3D II recognition using the multivariate statistical method

To evaluate the proposed 3D recognition system, we reconstruct 15 section images of sphacelaria alga and tribonema aequale alga corresponding to the reference data sets and the input data set by the II technique. The volumetric 3D image is reconstructed by the geometrical ray projection method [15]. We automatically remove the background parts of the reconstructed image by inspecting the histogram of the image and applying the Canny edge detection algorithm to the image data [29].

 

Fig. 5. The statistical distribution of the criterion discriminant function [see Eq. (3)] generated from the multi-dimensional data sets.

Download Full Size | PDF

We randomly select test pixel points from one well-focused section image, corresponding to the reference data, to form the reference M-ECDF, where 5000 pixel points were chosen. Then, we generate a multi-dimensional reference data set by selecting the pixel value of each section image at the same test pixel points. The statistical distribution of the criterion discriminant functions for the multi-dimensional reference data set is shown in Fig. 5, where we obtain F′_ref by selecting 500×15 random data samples distributed according to the reference M-ECDF and generate F′_ref 500 times to construct statistical sampling distribution of the criterion discriminant function Λˆ for the null hypothesis (F_ref =F_input ). As we expected, the value of the criterion discriminant function has a peak value at 0.5.

 

Fig. 6. The mean-square-distance (MSD) between the criterion discriminant function and the actual discriminant function [see Eq. (2)] generated from the multiple section images. Null hypothesis is the true training class. The data is obtained from 3D II volume images. (a) null hypothesis and (b) non-training true class and false class.

Download Full Size | PDF

 

Fig. 7. Results for one 2D image. The MSD between the criterion discriminant function and the actual discriminant function [see Eq. (2)] generated from one section image. Null hypothesis is the training true class. The data is obtained from 2D reconstructed image. (a) null hypothesis, (b) non-training true class, and (c) false class.

Download Full Size | PDF

To test the performance of the proposed 3D recognition system, we obtain the statistical distribution of the actual discriminant functions Λ for the multi-dimensional data sets of the true and false classes, respectively, according to similar procedures. The statistical distribution of the actual discriminant function Λ is formed by generating F_input 200 times and substituting the resulting values into Eq. (3), where this procedure can be repeated more often to obtain the precise statistical sampling distribution for the test statistics, Λ.

Figure 6 shows the statistical distribution of the MSD between the averaged criterion discriminant function <Λˆ(x ¹, …, x^p )> and the actual discriminant function Λ(x ¹, …,x^p ) of the true class and false classes, respectively. We obtain the statistical distribution of the MSD for the null hypothesis by calculating the MSD between <Λˆ(x ¹, …,x^p )> and Λˆ (x ¹, …, x^p ) to make a statistical decision rule. As shown in Fig. 6, the maximum value of the MSD for the null hypothesis is 0.0003 and the mean values of the MSD for the true and false class are 0.00026 and 0.0016, respectively. It is noted that all of the false classes over 200 trial data sets are above the maximum value of the MSD for the null hypothesis. To compare the value of the calculated MSD using only one section image (a well focused image), we also calculate the MSD between the averaged criterion discriminant function <Λˆ(x)> and the actual discriminant function Λ(x) using the data set that consisted of only one section 2D image.

Figure 7 shows the statistical distribution of the calculated MSD between <Λˆ(x)> and Λ(x) for the true and the false classes, respectively. As shown in Fig. 7, the maximum value of the MSD for the null hypothesis is 0.0033 and the mean values of MSD for the true and false classes are 0.0005 and 0.0006, respectively. It is noted that all of the 200 trial data sets for true and false classes are below the maximum value of the MSD for the null hypothesis. In this case, it is difficult to measure the similarity or dissimilarity between two data sets using only one section image. Thus, preliminary experimental results indicate that it may be possible to classify microorganisms using volumetric 3D images obtained by II and the multivariate statistical method.

5.2 3D II recognition using the morphology-based method

In this subsection, the RGM technique is applied to volumetric scenes using 3D grids. The 3D data is reconstructed using the II sensing and reconstruction method [15]. The parameters used for the Gabor-based wavelets were presented in Section 4. The Gabor-based wavelets are applied to the inversely normalized reconstructed images as discussed previously. The rectangular grids R and S are composed of 5×5×3 nodes and the distance between nodes is 10 pixels in both the x and y directions. In the z direction, the reference grid is superimposed on a reference image set, which is composed of 3 reconstructed images at depths of 324, 330, and 336 µm. Therefore, the number of depths (D) is 3, and the number of nodes in one image (K_grid ) is 25. The reference grid is placed with p _r=[116, 75]^t and θ_r =30° in each reconstructed image. The location and orientation of the reference grid is manually decided in the experiments.

 

Fig. 8. An example of RGM results. (a) A reference graph in the center image of the reference set and (b) the input graphs detected in the center image of the fifth input set.

Download Full Size | PDF

Figure 8 shows one reference image reconstructed at depth d=330 µm and one layer of the reference graph and the input graphs at the image. The threshold Γ _α in Eq. (12) is set at 0.995. The input graph is translated every two pixels in the x and y directions. To search the orientation angle, the input graph is rotated from 0° to 180° by 7.5° at every translated location, that is, the searching intervals for position and angle are set at 2 pixels and 7.5°, respectively. When the positions of the rotated nodes are not integers, they are replaced with the nearest neighbor nodes. We may improve the system performance by finer searching steps and/or high-order interpolation methods for the sake of computational complexity. However, as shown in the experimental results, the scheme presented in this paper can effectively recognize two different microorganisms illustrating the robustness of the system.

The tests are performed for 10 sets of reconstructed images for sphacelaria alga and tribonema aequale alga. The first set is composed of three images reconstructed at depths of 300, 306, and 312µm, and the second set is composed of images reconstructed at depths of 306, 312, and 318µm, and so on. Thus, the fifth set is identical to the reference image set for sphacelaria alga. Figures 9 and 10 show the maximum similarity function in Eq. (11) and the number of detections satisfying Eq. (12) for sphacelaria input image sets, respectively. As the reconstruction depths of the image set are closer to those of the reference images, more detections have been found showing strong similarity between the two image sets. No detection is found for any of the tribonema aequale alga input image set.

 

Fig. 9. Maximum similarity function for 10 input image sets of sphacelaria alga. Image set #5 is the referenced true class, others (1–4 and 6–10) are non-training true class image sets.

Download Full Size | PDF

 

Fig. 10. Number of detections and the maximum similarity function for 10 input image sets of sphacelaria alga. Image set #5 is the referenced true class, others (1–4 and 6–10) are nontraining true class image sets.

Download Full Size | PDF

7. Conclusions

In this paper, we have presented 3D identification of biological microorganisms by means of II. II records multiple views of a 3D scene with different perspectives by a single exposure. The volumetric information of the biological microorganism is reconstructed numerically by the ray projection method.

We have used volumetric 3D reconstruction using two identification methods. In one approach, we have estimated the statistical distributions of the biological microorganisms by the multivariate statistical method. The M-ECDF is computed using non-parametric methods for non-Gaussian distributed data. Then, we have calculated the MSD between the two discriminant functions to distinguish between the two data sets. We have presented experiments to measure the multivariate statistics obtained from II 3D data, and the results were compared with those obtained from 2D images.

In the morphology-based approach, the RGM technique is applied to multiple layers of reconstructed images using 3D grids. RGM is useful for identifying the reference shape in general cases that different species of microorganisms exist in the same sample, regardless of the curvature or shape of the microorganisms. We have shown that the extended RGM technique using 3D grids is suitable for investigating volumetric scenes reconstructed by the II system.