Focal stack camera in all-in-focus imaging via an electrically tunable liquid crystal lens doped with multi-walled carbon nanotubes

Hui Li; Jing Peng; Fan Pan; Yuntao Wu; Yanduo Zhang; Xiaolin Xie

doi:10.1364/OE.26.012441

1. Introduction

Nowadays, image processing and comprehension are the most essential fundaments of artificial intelligence (AI) technology in industrial manufacturing, computer vision, medicine, national defense, display, and entertainment fields [1–6]. More significantly, optical imaging is the important precondition of image processing and comprehension. To a certain extent, efficient optical imaging methods become solid cornerstone for AI technology. Despite major advances in optical imaging method, the conventional imaging way is still limited by some fundamental constraints. Generally, there is a significant sacrifice of image information. This is due to the dimension loss from 3D real scenes to 2D images. In order to remedy the loss of dimensionality reduction, many methods including hardware and software are proposed [7–9]. One of the most striking methods is to construct focal stack of a scene [10–12]. Department of Computer Science, Columbia University has done a lot of remarkable researches in this filed [13], so has Computer Science Department, Stanford University [14]. A focal stack is a sequence of 2D images taken at distinct focus for a scene. With the use of focal stack, all-in-focus image presenting the 3D scene information at details could be produced. In general, the algorithms to produce all-in-focus image from a focal stack are all algorithms of image fusion. There are approximately three categories, pixel level, feature level and decision level, respectively. Each of them has its own advantages [15–17]. But, image registration is a great challenge in common, and a huge amount of computation is necessarily required. That greatly limits the scope of this application. To solve this problem, a new efficient solution is urgently needed, especially the utilization of a novel optical imaging method.

In this paper, a focal stack camera, based on LC lens, is proposed. LC lenses is a novel type of electronically tunable optical imaging element. Because of LC's birefringence, this lens can form a gradient refractive index distribution by external electrically field. It has advantages over liquid lenses, such as no gravity effect, more compact without complex driving device, and no reflection from the liquid–liquid interface [18,19]. The proposed focal stack camera based on LC lens technique differs from the light field camera in one aspect. Because the focal stack produced by a light field camera or a plenoptic camera just records a single instant of time for a scene, the resolution of each reconstructed 2D images in the focal stack is not very high [20]. However, a focal stack camera captures 2D images with distinct focusing settings directly. In this way, sensor spatial resolution for every captured 2D images in the focal stack can be preserved [21]. In addition, there are several major reasons for choosing LC lens instead of ordinary glass lenses. The LC lens is relatively small size and can be tuned by external electrical field [22–25]. Because of the LC lens imaging without mechanical jitter, image registration is not a main concern. The significant amount of computation power and time for image registration are not needed. For LC lens, the depth map is not necessary because the depth information clue implies on the series of 2D images in the focal stack. The optical performance of the LC lens is not limited by the fundamental constraints among aperture, depth of field (DOF), exposure time, and exposure level that restrict traditional glass lenses [26, 27]. Its image quality is less affected by the above mentioned factors.

Owing to its electrically tunable focusing, compactness, portability, and easy integration with other optical devices, LC lenses, in many imaging applications, is an effective alternative to traditional optical glass lenses [28–30]. However, there are still some limitations when fabricating high quality LC lenses, such as slow response time and not high image quality [28–30]. Previous studies have reported that the doping technology is an effectively method of improving LC device performance [31–33]. In this study, a grinding method is utilized to lower agglomeration phenomenon, and the nano doping method is adopted to improve electro-optical features of LC lens. A LC lens doped with multi-walled carbon nanotubes (MWCNTs) is devised. An efficient depth range sampling strategy is realized by adopting the LC lens. And a relatively simple algorithm combined with the focal stack camera based on LC lens is proposed to generate a single all-in-focus image.

Section 2 presents the theories of focal stack camera built with LC lens, focusing on its DOF and its focal sweep speed, and includes an analysis of a relatively simple algorithm to produce all-in-focus image with proposed focal stack camera. Section 2 also presents the fabrication of LC lens doped with MWCNTs. The experimental results about the classic optical features of the proposed LC lens are presented in Section 3. Those interesting results about producing all-in-focus image based on the LC lens are also discussed and analyzed at details. Finally, the conclusion gives a brief summary about the presented method of producing a single all-in-focus image.

2. Operating principle and sample preparation

2.1 Focal stack camera based on LC lens

Focal stack camera is a novel conception to solve dimensionality reduction of imaging in a scene [13,14]. The key structure of the focal stack camera consists of a LC lens and a camera with a fixed focal length lens, as shown in Fig. 1. With the use of LC lens, the camera imaging jitter can be substantially reduced. Capturing a series of well-lit 2D images, the focal stack could be constructed. With overall considerations about total capture time, camera frame rate, and images quality, a focal sweep strategy is devised. As a matter of fact, the focal sweep strategy of the focal stack camera is the strategy of the LC lens. For the strategy, the DOF and focal sweep speed of the LC lens-based focal stack camera are the most important keys, as shown in Fig. 1. In the focal stack, every 2D image has one major focused area for the scene. To sample a desired depth range, the whole DOFs of all captured 2D images should ideally cover the entire desired depth range, and the end of one 2D image's DOF is better to be the start of the next 2D image's DOF [34].

Fig. 1 Efficient and complete focal sweep strategy.

Download Full Size | PDF

1) DOF of the focal stack camera

In Fig. 2, schematic diagram of the proposed focal stack camera is presented. The distance between LC lens and the fixed focal length lens is a fixed value, L. There are two sub-image systems. The first image is produced by LC lens. And the second image is generated by the fixed focal length lens. To express conveniently, the mathematical symbols are presented as follows. Points 1, 2, and 3 denote different positions of the same object in a scene. Correspondingly, m₁, m₂, and m₃ present the object distances for the same object, respectively. Points 1', 2', and 3′ are the first images through the LC lens. Point 2' is just on the image plane. Then, p₁, p₂, and p₃ mean the image distances after the first images. For the fixed focal length lens, q₁, q₂, and q₃ are the object distances. Points 1”, 2”, and 3” are the second images through the fixed focal length lens. Point 2” is just on the image plane at CCD. Point 1” and 3” are not on the image plane. But, they are precisely at the border between clear image and blurred image. And n₁, n₂, and n₃ mean the image distances after the second image.

Fig. 2 DOF of the proposed focal stack camera.

Download Full Size | PDF

According to the Thin Lens Law, $1 / f = 1 / u + 1 / v$ , where f is the focal length of a lens, u is the object distance, and v is the image distance [35]. When the reciprocal is introduced, $\hat{x} = 1 / x$ , the Thin Lens Law can be transformed to $\hat{f} = \hat{u} + \hat{v}$ . In the reciprocal domain, the Thin Lens Law becomes a linear equation.

Defocus blur is a consequence of geometric optics. As shown in Fig. 2, a blur spot of radius c_LC forms on the image plane. This blur spot is known as the circle of confusion, and its shape is referred to the LC lens' bokeh. Considering similar triangle, $Δ_{L C} = c_{L C} \cdot p / r_{L C}$ , where $Δ_{L C}$ is the distance between the image plane and the bokeh plane, and r_LC denotes the radius of the top circular electrode pattern in the proposed LC lens. Similarly, for the fixed focal length lens, $Δ_{L} = c_{L} \cdot p / r_{L}$ , where $Δ_{L}$ is the distance between the image plane and the bokeh plane, and r_L denotes the radius of its aperture. For the LC lens, the objects at Points 1, 2, and 3 forms images, respectively. Suppose $m_{1} = m_{2} - Δ_{L C}$ , and $m_{3} = m_{2} + Δ'_{L C}$ , then,

D O F = | m_{1} - m_{3} | = 2 \cdot | p_{3} - p_{1} | \cdot \frac{c_{L C}}{r_{L C}} .

For the fixed focal length lens, the imaging formulas are respectively,

{\hat{f}}_{L} = {\hat{n}}_{1} + {\hat{q}}_{1},

{\hat{f}}_{L} = {\hat{n}}_{2} + {\hat{q}}_{2},

{\hat{f}}_{L} = {\hat{n}}_{3} + {\hat{q}}_{3} .

Where $n_{1} = n_{2} - Δ_{L}$ , and $n_{3} = n_{2} + Δ_{L}$ . Equation (4) subtracts Eq. (2), and gets its absolute value,

| {\hat{q}}_{3} - {\hat{q}}_{1} | = | {\hat{n}}_{3} - {\hat{n}}_{1} | .

Because of the reciprocal domain, both sides of Eq. (5) are multiplied by $q_{3} \cdot q_{1}$ ,

| q_{3} - q_{1} | = q_{3} \cdot q_{1} \cdot | {\hat{n}}_{3} - {\hat{n}}_{1} | .

For the left of Eq. (6), a deduction is as following, $| q_{3} - q_{1} | = | q_{3} - q_{1} + p_{3} - p_{1} - (p_{3} - p_{1}) |$ . As $L = p_{1} + q_{1} = p_{3} + q_{3}$ , $| q_{3} - q_{1} | = | p_{3} - p_{1} |$ . Correspondingly, Eq. (6) can be expressed as,

| p_{3} - p_{1} | = q_{3} \cdot q_{1} \cdot | {\hat{n}}_{3} - {\hat{n}}_{1} | .

Therefore, under one applied voltage, the DOF of the LC lens-based focal stack camera is, $D O F = | m_{1} - m_{3} |$ $= 2 \cdot \frac{q_{3} \cdot q_{1} \cdot c_{L C}}{n_{3} \cdot n_{1} \cdot r_{L C}} | n_{3} - n_{1} |$ . That means the DOF of the LC lens-based focal stack camera can be tunable by the external applied voltage. In the focal stack, the end of one 2D image's DOF can be adjusted to the start of the next 2D image's DOF.

2) Focal sweep speed of the focal stack camera

In the focal stack camera, LC lens is an electrically tunable optical element. The focal length is a variable with the change of applied voltage. The schematic diagram of focal sweep speed of the LC lens is as shown in Fig. 1. According to the Thin Lens Law, under different applied voltages, LC lens has,

\frac{1}{f_{L C} (V_{1})} = \frac{1}{m} + \frac{1}{p},

\frac{1}{f_{L C} (V_{2})} = \frac{1}{m'} + \frac{1}{p'} .

Where $f_{L C} (V_{1})$ means the focal length of the LC lens at V₁, m denotes the object distance, and p is the image distance. Similarly, $f_{L C} (V_{2})$ , m', and p' are parameters at V₂. With the use of reciprocal domain, Eq. (8) can be expressed as, ${\hat{f}}_{L C} (V_{1}) = \hat{m} + \hat{p}$ , and Eq. (9) can be also expressed as ${\hat{f}}_{L C} (V_{2}) = \hat{m}' + \hat{p}'$ .

The above two equations are subtracted from each other, then get the absolute value,

| {\hat{f}}_{L C} (V_{2}) - {\hat{f}}_{L C} (V_{1}) | = | \hat{m}' - \hat{m} + \hat{p}' - \hat{p} | .

If $Δ f = | f_{L C} (V_{1}) - f_{L C} (V_{2}) |$ , $Δ m = | m' - m |$ , and $Δ p = | p' - p |$ , $Δ f$ is,

Δ f = f_{L C} (V_{1}) \cdot f_{L C} (V_{2}) \cdot | \frac{Δ m}{m' \cdot m} + \frac{Δ p}{p' \cdot p} | .

Under extreme conditions, $m ≫ p$ , $p \approx f_{L C \min}$ , $p' \approx f_{L C \max}$ . By approximating, Eq. (11) has,

Δ f \geq \frac{f_{L C \min}}{f_{L C \max}} \cdot | f_{L C \min} - f_{L C \max} | .

Equation (12) suggests that an efficient and complete sampling speed can change the focal length of LC lens by a constant value between every consecutive 2D image captures. The changing speed is determined by the minimum and maximum values of the LC lens. Notice that this is a constant step in the normal domain, rather than in the reciprocal domain. Hence, if the LC lens operates at a constant frame-rate, the ideal strategy to sample the desired depth range is to electronically control the sampling rate at a constant speed.

3) Algorithm of producing all-in-focus image

To produce a single all-in-focus image, a proper image fusion algorithm is required for the focal stack camera. As there are no image registration steps for all 2D images captured by LC lens in the focal stack, the relatively simple average algorithm is a proper method for producing all-in-focus image. However, this method usually causes the halo effects. In this section, an improved algorithm based on the average method is given.

In the focal stack, there has been a sequence of 2D images taken by LC lens under different applied voltages. Those 2D images are captured by the proposed focal sweep strategy. The focal stack presenting as $I_{k}$ , k = 1,…, M, M means the number of 2D images. When the image signal reaches a local extremum, the Laplacian of this position reaches a zero value. The intensity changes fastest in the neighbor of this pixel. With the use of Laplacian, one 2D image where some particular pixels are sharp can be distinguished from the rest of 2D images in the focal stack. This transform in the sharp images is smaller and closer to zero than in the areas in the blurred images. Correspondingly, image gradient describing value in sharp regions change rapidly and the blurred regions change slowly.

The improved algorithm flow of producing all-in-focus image (see Fig. 3) is as following.

Fig. 3 The pipeline of the image fusion algorithm for producing all-in-focus image.

Download Full Size | PDF

1. The weight value for every 2D image is defined as the coefficient of the Laplacian transform and image gradient, $a_{k} (i, j) = \frac{| G_{k} (i, j) |}{| L_{k} (i, j) | + ε}, k = 1, \dots, M .$
Where, G_k means image gradient of every images in the focal stack, L_k is the Laplacian transform, k = 1,…, M, i and j represents the horizontal and vertical pixel indexes of images, and $ε$ stands for a small number to prevent zero division.
2. With the use of the weighted average function, all-in-focus image can be produced, $I (x, y) = \frac{\sum_{i} a_{k} (x, y) \cdot I_{k} (x, y)}{\sum_{i} a_{k} (x, y) + ε} .$
With this strategy, severely blurred patches will have much less weight than sharper patches do, reducing the hazy effects that are often observed in the averaged image method in the conventional image fusion. The weighted sum is sharp and has high contrast even without frequency-domain-transformation. And it can avoid the risk of producing frequency-domain-transformation artifacts and reduce the halo effects introduced by average accumulation.

2.2 Sample preparation

To fabricate the LC lens illustrated in Fig. 4, MWCNTs (from Shenzhen Nanotech Port Co. Ltd, > 90 wt.%, a length of 1 – 2 μm, and an outside diameter of 10 – 20 nm) are adopted. The LC lens structure consists of two glass substrates, indium tin oxide (ITO) layers serving as electrodes, a highly resistive PEDOT layer, alignment layers of polyimide (PI), and LC layer doped with MWCNTs. To reduce the agglomeration, those doped MWCNTs are required to process as follows. MWCNTs were dried in near vacuum at 110°C for 4 hours. After cooling to room temperature, MWCNTs were grinded by a powder grinding machine. The grinding procedure repeated for 3 more times, and each time lasted 30 seconds against overheating. Nematic LC (Merck, E7) and processed MWCNTs with a 99.98:0.02 wt.% ratio were as a mixture. In a clean glass-tube, the mixture was continually shocked by an ultrasonic shaker for 3.5 hours. After that, a test tube shaker further mixed the mixture for another 30 minutes at 1000 rmp/m. Then, the mixture controlled at 80°C to be isotropic was filled between ITO glass substrates coated with mechanically buffered polyimide (PI) with parallel rubbing directions along x-axis by capillarity. In addition, a highly resistive PEDOT (from Sigma-Aldrich) layer fabrication process is as follows. A 30nm thickness of high resistance layer was produced by 4000rpm/s for 30 seconds. This was a proper value to avoid becoming a conducting layer or increasing the driving voltage. By measurement, its surface resistance was about 1M $Ω$ /□. For stabilization, this layer required twice heating. First, it was set at 120°C for 1 hour. Second, it was set at 150°C lasting 1.5 hours. Then, annealing method was utilized to prevent damage to this high resistance layer. It was set at 150°C. The whole time from set-temperature to room-temperature was about 2 hours. In addition, the important parameters for fabricating the LC lens were as follows. A voltage with the change from 0Vrms to 20Vrms at a frequency of 1 kHz was applied to the ITO glass substrates for driving. The thickness of LC layer was 100μm. The top ITO layer in Fig. 4 was etched with a 2.3mm-diameter hole, which defined the LC lens aperture size.

Fig. 4 The structure of the proposed LC lens.

Download Full Size | PDF

3. Results and Discussions

In the proposed focal stack camera, the LC lens and a polarizer were attached to a camera, as shown in Fig. 1. The camera consists of 1) an imaging lens with an effective fixed focal length 25mm, and 2) a CCD as an image sensor with 3 Megapixel, 1/2 inch, and 6.4mm × 4.8mm in size. The rubbing direction of the LC lens has a 45 degree angle with a polarizer. The LC lens was driven by a function generator with a tunable applied voltage at 1kHz frequency. During the whole measurement, both the LC lens and the imaging lens were fixed without any mechanical movements.

To evaluate the performances of the proposed focal stack camera, the classic optoelectronic properties of the LC lens are firstly measured. In consideration of LC birefringence, the tunable focusing properties are the major advantage of the LC lens. To observe the phase profile of the LC lens, the interference-pattern images of the LC lens at different applied voltage under crossed polarizers are as shown in Figs. 5(a) and 5(b). The focal length formula of the LC lens is $f = \frac{w^{2}}{8 \cdot N \cdot λ}$ , where w means diameter of the circular electrode pattern on the top substrate, N is the number of interference rings, and $λ$ is the wavelength of the incident laser [36]. The rubbing direction of the LC lens was 45° with respect to one polarizer. To measure the lens power of the LC lens, USAF 1951 was set in front of the camera. A spatial frequency of 32.0lp/mm was chosen in the resolution chart in order to measure its image details. When the applied voltage of the LC lens was changed to a value, the distance between the LC lens and the imaging sensor was immediately adjusted until the image of the resolution chart was clear again. With this subjective judgment method, the focal lengths of the LC lens at different applied voltages were recorded. The lens power was calculated by converting the acquired focal length. In this study, 1m denotes 1Diopter (D). The measured lens power of the LC lens as a function voltage is presented in Fig. 5(c). When the external applied voltage is adjusted from 0Vrms to 5.0Vrms, the lens power of the LC lens is capable of switching from 1.922D to 83.33D. Compared to the conventional LC lens (the same structure, especially the LC lens without dopant, just from 2.08D to 50.0D), the doped dopant has great improvements on the properties of the LC lens. When MWCNTs are doped into LC, various ions could be absorbed by MWCNTs because of its large surface area. Absorbing mainly happens at inner surfaces in contact with the LC of the substrates in the LC lens. As a result, the dispersion obviously decreases [37, 38]. In addition, metal property of MWCNTs could help LC molecular efficiently align orderly under the external applied voltage [37, 38]. With the use of doped MWCNTs, the range of focal length is effectively extended.

Fig. 5 Classic optoelectronic properties of the LC lens. (a) is the phase profile of the LC lens at 1.5Vrms; (b) is the phase profile of the LC lens at 2.5Vrms; (c) The lens power of the LC lens as the function of the applied voltage, those red circular dots mean the LC lens doped with MWCNTs, and those blue square dots mean the conventional LC lens without any dopant.

Download Full Size | PDF

To present the response time of the LC lens doped with MWCNTs, the switching time from one focusing state to another focusing state was firstly recorded. Table 1. presents the measurement results. Without doped MWCNTs, the switching time of the conventional LC lens was 2.6s. When the MWCNTs were doped, the response time reduced to 0.1s. This means the dopant can effectively improve the performance of the LC lens. The switching time is very important for constructing the focal stack, especially on the focal sweep strategy. For measuring the operation time of LC lens, the starting time and the end time were defined as the time points where the transmitted light power increased to 10% and 90% of the initial value under an external applied voltage respectively. Measured by EOT-01, the operation voltage of the conventional LC lens was generally above 20.0Vrms, and its operation time was about 20s. In contrast, the operation voltage of the proposed LC lens was below 5.0Vrms, and its operation time was only 2s [37, 38]. The results show that the proposed LC lens shortened its operation time and was driven by a relatively low operation voltage compared to the conventional one. Overall, the proposed LC lens dramatically improves about 90% of the operation time and reduces about 75% of the operation voltage. In despite of the above improvements, the LC lens doped with MWCNTs is still not as fast as liquid lens on the response time [18, 19]. However, the LC lens can provide much better image performance and easy fabrication process compared to the liquid lens [18, 19]. With comprehensive consideration about cost, fabrication difficulty, response time, and image quality, LC lens is very suitable to utilize for the focal stack camera.

Table 1. Comparison results about switching time between the LC lens doped with MWCNTs and the conventional LC lens^a

View Table | View all tables in this article

To demonstrate the proposed focal stack camera, a 3D scene includes three objects placed in front of the LC lens, the first hello kitty at 1.5cm, the second one 3.0cm, and the third one 4.5cm, respectively. To apply the focal sweep stagey in Section 2.1, 2D images could be taken directly by the LC lens. The focal sweep strategy was to use the proposed LC lens with a constant speed of loading the external driven applied voltage. The switching time from one focusing state to another state is nearly a constant, 100ms. The applied voltage increased by 0.5Vrms from 0Vrms to 5Vrms in turn. 10 frames were got to construct the focal stack in only one second. The captured 2D images are as shown in Fig. 6. In Fig. 6, the LC lens at 1.0Vrms has a little lens power, and only the hello kitty at 4.5cm is clearly imaged while others are blurred due to being outside of the DOF. In Fig. 6, the hello kitty at 3.0cm is in focus within the DOF as the LC lens is at 2.5Vrms. At 4.0Vrms, the hello kitty at 1.5cm is clearly imaged within the DOF, as shown in Fig. 6. Those captured 2D image in Fig. 6 even local magnification still have good imaging contrast. From those local magnification patches, there are clear image details. Generally, the conventional LC lens has a normal imaging contrast. The main reason about the improvement in imaging contrast is ions effect in doped LC. When MWCNTs are doped into LC, various ions could be dramatically absorbed by MWCNTs. The majority of absorbing happens at inner surfaces in contact with the LC of the substrates in the LC lens. Under this condition, the ionic concentration significantly reduces [37, 38].

Fig. 6 Three focused states in the focal stack based on the LC lens. (c) is at 1.0Vrms; (b) is at 2.5Vrms; (a) is at 4.0Vrms.

Download Full Size | PDF

In the focal stack, there are a series of acquired 2D images with different applied voltages at various object distances. As it is known, there are so many different kinds of focus measure functions [39]. In this study, 3 × 3 Sobel operator, $G (k) = \sum_{x = 1}^{M - 2} \sum_{y = 1}^{N - 2} (| G_{x} | + | G_{y} |)$ , is chosen as the focus measure function, where $G_{x} = [\begin{matrix} 1 & 0 & - 1 \\ 2 & 0 & - 2 \\ 1 & 0 & - 1 \end{matrix}]$ , and $G_{y} = [\begin{matrix} 1 & 2 & 1 \\ 0 & 0 & 0 \\ - 1 & - 2 & - 1 \end{matrix}]$ . Figure 7 shows the Sobel operator as a focus measure function under the external applied voltage when the LC lens aims at the different objects in a scene respectively. As it is seen from Fig. 7, when the external applied voltage at 4.0Vrms, the measure function has the highest value for the first hello kitty, at 2.5Vrms for the second hello kitty, and at 1.0Vrms for the third hello kitty. Those indicate that the capture focused images in the focal stack have different focused regions when the object was located between 12mm and 520mm by changing the driven operate voltages. The focus measure function is utilized to evaluate the focus degree of the LC lens at various applied voltages. From those processed 2D images in the focal stack, the focused parts are different form the unfocused parts. Clear edges in those 2D images show the focused regions. Equation (13) is the formula of weight value for every 2D image in the focal stack. It is defined as a coefficient of Laplacian transform and image gradient. Figure 8 shows the weight value (firstly absolute value, then normalization) as a function of applied voltage. There are obviously three peaks, 1.0Vrms, 2.5Vrms, and 4.0Vrms. Those mean that there is a clear focused object at every peak. Every peak is corresponding to its imaging state at a voltage value. To find out the desired DOFs, a resolution char (USAF 1951) with a spatial frequency of 32.0lp/mm was utilized as an object placed in the above scene. This object at different distance was imaged on CCD. The distance was adjusted along the optic axis until the image of the resolution char was detailed as highly as the degree at the corresponding peak state. The criterion for discrimination is comparing the calculated value (by Eq. (13)) of measured image with the calculated value (by Eq. (13)) of peak image. The closest value is the final measurement. In this way, DOFs under different voltages were acquired. Table 2. shows the specification of the focal stack camera based on the proposed LC lens. From those data, it can be found that the whole DOFs of all captured 2D images could cover the entire desired depth range. That's just coincides with the focal sweep strategy in Section 2.1.

Fig. 7 The focus measure function of 2D images in the focal stack at different applied voltages. (a) is at 0.5Vrms; (b) is at 1.0Vrms; (c) is at 1.5Vrms; (d) is at 2.0Vrms; (e) is at 2.5Vrms; (f) is at 3.0Vrms; (g) is at 3.5Vrms; (h) is at 4.0Vrms; (i) is at 4.5Vrms; (j) is at 5.0Vrms.

Download Full Size | PDF

Fig. 8 The weight value as a function of applied voltage, which is calculated for every 2D image in the focal stack.

Download Full Size | PDF

Table 2. The specification of the focal stack camera based on the proposed LC lens ^a

View Table | View all tables in this article

To further demonstrate the advantages of LC lens in generate all-in-focus image, Power Signal-to-Noise Ratio (PSNR) and Root Mean Square Error (RMSE) of the 2D images in the focal stack were introduced [40]. As the LC lens was adopted, both focused and defocused regions exist in the captured 2D images. For traditional optical systems, the PSNR and RMSE will have bigger changes when the focal length of the system is adjusted. In addition, there are also still some factors to influence image quality between the series of 2D images in the stack, such as ambient light, and mechanical jitter. In this way, the focal stack based on the conventional method has bigger changes between the series of 2D images, especially on the edge of those 2D images. This is the main reason for an amount of calculation to process image registration problem. Generally, image registration is a huge problem for image fusion, especially in generating all-in-focus image. In image processing algorithm domain, that is an unavoidable problem. To measure quantitatively variations between the series of 2D images in the focal stack, the PSNR and RMSE were calculated as a similarity measure. 2D image at 0.5Vrms was chosen as a contrasting baseline image. The other 2D images in the focal stack were computed PSNR and RMSE with the baseline image. As those 2D images chosen from the focal stack have slight changes, the values of PNSR and RMSE have little differences, as presented in Fig. 9. There is just 3% change in PNSR. And there is just 5% change in RMSE. With very limited variations between the series of 2D images in the focal stack, LC lens can completely avoid the requirement of image registration step.

Fig. 9 PSNR and RMSE for 2D images in the focal stack at different applied voltages.

Download Full Size | PDF

To further demonstrate the algorithm of generating all-in-focus image, the weight average method mentioned in Section 2.2 was utilized. For this image fusion, the conditions of the computing equipment are as follows, CPU i3-2100, and 8G RAM. The whole time for producing a single all-in-focus image contains the captured time to construct the focal stack and the image fusion time. The previous time was just ten times the single focusing time. The latter one was just 1s. The total time was about 2s. That is much faster than some conventional algorithms for image fusion in generating all-in-focus image. In Fig. 10, the discrete cosine transform (DCT) method, the average method, and the weighted average method are presented. All Figs. 10(a)–10(c) preserve most high frequency information. The average image yields a low contrast, especially when the number of images increases. The DCT method is prone to image artifacts. In addition, the DCT method is computationally expensive. With the same computing equipment, this DCT method is about 20s. That is much longer than the proposed method. In this study, a weighted method was utilized. With the strategy mentioned in the Section 2.2, severely blurred patches will have much less weight than sharper patches do, reducing the hazy effects that are often observed in the average image from Fig. 10(b). As shown in Fig. 10(c), the weighted sum is sharp and has high contrast. In Fig. 10(d), there are the best focused patches in the captured focal stack. Compared those patches with Fig. 10(c), the image quality is nearly the same, and all images are at sensor resolution. In this way, this weighted method could avoid the risk of producing image artifacts. From Fig. 10, the results show that the proposed method to generate all-in-focus image has not only relatively fast speed but also a high resolution, even at sensor resolution.

Fig. 10 All-in-focus image computed using different approaches and their close-ups. (a) The DCT method to fuse all 2D images in the focal stack; (b) The average method to fuse all 2D images in the focal stack; (c) The weighted average method to fuse all 2D images in the focal stack; (d) The best focused patches in the captured focal stack.

Download Full Size | PDF

4. Conclusions

In this paper, a method of generating all-in-focus image based on an electrically tunable LC lens is proposed. Its core optical imaging element is LC lens. As a result of reduction of ionic effect, the proposed LC lens has high performances on fast response time and high resolution imaging quality. Those factors were very important for producing a single all-in-focus image. An algorithm of weight average method was utilized as a best solution for the focal stack camera. Because of adopting LC lens, the time for generating a single all-in-focus image is much faster than the other conventional focal stack methods, especially in its fast response time and image fusion time without image registration. For producing a single all-in-focus image, the proposed LC lens with the devised focal sweep strategy is maximize utilized. The result of all-in-focus image has sensor spatial resolution compared to some conventional focal stack methods. The proposed method has a great potential in a compact 3D imaging system for AI applications. However, the presented result about generating all-in-focus image is only prototype, and there is still a room for further developing in future, given how to further improve imaging quality by eliminating the polarization of LC lens.

Funding

National Natural Science Foundation of China (51703071); Education Bureau of Hubei Province China (D20171504, B2016489); China Postdoctoral Science Foundation (2014M562017); Wuhan Institute of Technology (Q201601); Hubei Key Laboratory of Intelligent Robot (HBIR 201705).

Acknowledgments

We will give our thanks to Sanrio Co., LTD. for creating such a vivid cartoon character (hello kitty) and permission for publication in OSA journals.

References and links

1. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications [Invited],” Appl. Opt. 52(4), 546–560 (2013). [CrossRef] [PubMed]

2. J. Geng, “Three-dimensional display technologies,” Adv. Opt. Photonics 5(4), 456–535 (2013). [CrossRef] [PubMed]

3. J. Xi, L. Shentu, J. Hu, and M. Li, “Automated surface inspection for steel products using computer vision approach,” Appl. Opt. 56(2), 184–192 (2017). [CrossRef] [PubMed]

4. E. J. Fernández, B. Hermann, B. Považay, A. Unterhuber, H. Sattmann, B. Hofer, P. K. Ahnelt, and W. Drexler, “Ultrahigh resolution optical coherence tomography and pancorrection for cellular imaging of the living human retina,” Opt. Express 16(15), 11083–11094 (2008). [CrossRef] [PubMed]

5. S. H. Hong, J. S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express 12(3), 483–491 (2004). [CrossRef] [PubMed]

6. C. Huang, Q. Zhang, H. Wang, and S. Feng, “A low power and low complexity automatic white balance algorithm for AMOLED driving using histogram matching,” J. Disp. Technol. 11(1), 53–59 (2015). [CrossRef]

7. E. Y. Lam, “Computational photography with plenoptic camera and light field capture: tutorial,” J. Opt. Soc. Am. A 32(11), 2021–2032 (2015). [CrossRef] [PubMed]

8. A. Castro and J. Ojeda-Castañeda, “Asymmetric phase masks for extended depth of field,” Appl. Opt. 43(17), 3474–3479 (2004). [CrossRef] [PubMed]

9. S. W. Hasinoff and K. N. Kutulakos, “Light-efficient photography,” IEEE Trans. Pattern Anal. Mach. Intell. 33(11), 2203–2214 (2011). [CrossRef] [PubMed]

10. C. Liu, J. Qiu, and M. Jiang, “Light field reconstruction from projection modeling of focal stack,” Opt. Express 25(10), 11377–11388 (2017). [CrossRef] [PubMed]

11. X. Yin, G. Wang, W. Li, and Q. Liao, “Iteratively reconstructing 4D light fields from focal stacks,” Appl. Opt. 55(30), 8457–8463 (2016). [CrossRef] [PubMed]

12. A. Koppelhuber, C. Birklbauer, S. Izadi, and O. Bimber, “A transparent thin-film sensor for multi-focal image reconstruction and depth estimation,” Opt. Express 22(8), 8928–8942 (2014). [CrossRef] [PubMed]

13. C. Zhou, D. Miau, and S. K. Nayar, “Focal sweep camera for space-time refocusing, ” http://www1.cs.columbia.edu/CAVE/projects/focal_sweep_camera/.

14. D. E. Jacobs, J. Baek, and M. Levoy, “Focal stack compositing for depth of field control,” https://graphics.stanford.edu/papers/focalstack/.

15. X. Yan, H. Qin, J. Li, H. Zhou, and T. Yang, “Multi-focus image fusion using a guided-filter-based difference image,” Appl. Opt. 55(9), 2230–2239 (2016). [CrossRef] [PubMed]

16. L. Chen, J. Li, and C. L. Chen, “Regional multifocus image fusion using sparse representation,” Opt. Express 21(4), 5182–5197 (2013). [CrossRef] [PubMed]

17. L. Guo, M. Dai, and M. Zhu, “Multifocus color image fusion based on quaternion curvelet transform,” Opt. Express 20(17), 18846–18860 (2012). [CrossRef] [PubMed]

18. R. Peng, J. Chen, and S. Zhuang, “Electrowetting-actuated zoom lens with spherical-interface liquid lenses,” J. Opt. Soc. Am. A 25(11), 2644–2650 (2008). [CrossRef] [PubMed]

19. H. Ren, D. Fox, P. A. Anderson, B. Wu, and S. T. Wu, “Tunable-focus liquid lens controlled using a servo motor,” Opt. Express 14(18), 8031–8036 (2006). [CrossRef] [PubMed]

20. Y. Jeong, J. Kim, J. Yeom, C. K. Lee, and B. Lee, “Real-time depth controllable integral imaging pickup and reconstruction method with a light field camera,” Appl. Opt. 54(35), 10333–10341 (2015). [CrossRef] [PubMed]

21. F. Perez, A. Perez, M. Rodriguez, and E. Magdaleno, “A fast and memory-efficient discrete focal stack transform for plenoptic sensors,” Digit. Signal Process. 38, 95–105 (2015). [CrossRef]

22. Y. H. Lin, M. S. Chen, and H. C. Lin, “An electrically tunable optical zoom system using two composite liquid crystal lenses with a large zoom ratio,” Opt. Express 19(5), 4714–4721 (2011). [CrossRef] [PubMed]

23. M. S. Chen, P. J. Chen, M. Chen, and Y. H. Lin, “An electrically tunable imaging system with separable focus and zoom functions using composite liquid crystal lenses,” Opt. Express 22(10), 11427–11435 (2014). [CrossRef] [PubMed]

24. Y. Lin and M. Chen, “A pico projection system with electrically tunable optical zoom ratio adopting two liquid crystal lenses,” J. Disp. Technol. 8(7), 401–404 (2012). [CrossRef]

25. H. C. Lin, N. Collings, M. S. Chen, and Y. H. Lin, “A holographic projection system with an electrically tuning and continuously adjustable optical zoom,” Opt. Express 20(25), 27222–27229 (2012). [CrossRef] [PubMed]

26. H. S. Chen and Y. H. Lin, “An endoscopic system adopting a liquid crystal lens with an electrically tunable depth-of-field,” Opt. Express 21(15), 18079–18088 (2013). [CrossRef] [PubMed]

27. Y. J. Wang, X. Shen, Y. H. Lin, and B. Javidi, “Extended depth-of-field 3D endoscopy with synthetic aperture integral imaging using an electrically tunable focal-length liquid-crystal lens,” Opt. Lett. 40(15), 3564–3567 (2015). [CrossRef] [PubMed]

28. S. Xu, Y. Li, Y. Liu, J. Sun, H. Ren, and S. T. Wu, “Fast-response liquid crystal microlens,” Micromachines (Basel) 5(2), 300–324 (2014). [CrossRef]

29. H. Ren, S. Xu, Y. Lin, and S. Wu, “Adaptive-focus lenses,” Opt. Photonics News 19(10), 42–47 (2008). [CrossRef]

30. H. C. Lin, M. S. Chen, and Y. H. Lin, “A review of electrically tunable focusing liquid crystal lenses,” Trans. Electr. Electron. Mater. 12(6), 234–240 (2011). [CrossRef]

31. S. Y. Lu and L. C. Chien, “Carbon nanotube doped liquid crystal OCB cells: physical and electro-optical properties,” Opt. Express 16(17), 12777–12785 (2008). [CrossRef] [PubMed]

32. W. Lee and C. S. Chiu, “Observation of self-diffraction by gratings in nematic liquid crystals doped with carbon nanotubes,” Opt. Lett. 26(8), 521–523 (2001). [CrossRef] [PubMed]

33. R. Rajasekharan-Unnithan, H. Butt, and T. D. Wilkinson, “Optical phase modulation using a hybrid carbon nanotube-liquid-crystal nanophotonic device,” Opt. Lett. 34(8), 1237–1239 (2009). [CrossRef] [PubMed]

34. K. Kutulakos and S. Hasinoff, “ Focal stack photography: High-performance photography with a conventional camera, ” in Proc. 11th IAPR Conference on Mach. Vision Appl. 332–337 (2009).

35. F. Jenkins and H. White, Fundamentals of Optics (McGraw-Hill Education, 2001).

36. H. Ren and S. T. Wu, Introduction to Adaptive Lenses (John Wiley, 2012).

37. H. Li, F. Pan, Y. Wu, Y. Zhang, and X. Xie, “Improvement in imaging contrast feature of liquid crystal lens with the dopant of multi-walled carbon nanotubes,” Appl. Opt. 56(23), 6655–6662 (2017). [CrossRef] [PubMed]

38. L. Hui, P. Fan, W. Yuntao, Z. Yanduo, and X. Xiaolin, “Depth map sensor based on optical doped lens with multi-walled carbon nanotubes of liquid crystal,” Appl. Opt. 55(1), 140–147 (2016). [CrossRef] [PubMed]

39. P. Pavliček and I. Hamarová, “Shape from focus for large image fields,” Appl. Opt. 54(33), 9747–9751 (2015). [CrossRef] [PubMed]

40. R. C. Gonzalez and R. E. Woods, Digital Image Processing (Pearson, 3 edition, 2007).

Comparison samples	Doping concentration (wt.%)	Switching time (s)
The proposed LC lens	0.02 wt.%	0.1sec
Conventional LC lens	0.0 wt.%	2.6sec

Applied voltage (Vrms)	The closest distance of acceptable sharpness (mm)	The farthest distance of acceptable sharpness (mm)	The distance of an object (mm)	DOF (mm)
1.0Vrms	439mm	464mm	450mm	25mm
2.5Vrms	285mm	315mm	300mm	30mm
4.0Vrms	130mm	172mm	150mm	42mm

Comparison samples	Doping concentration (wt.%)	Switching time (s)
The proposed LC lens	0.02 wt.%	0.1sec
Conventional LC lens	0.0 wt.%	2.6sec

Applied voltage (Vrms)	The closest distance of acceptable sharpness (mm)	The farthest distance of acceptable sharpness (mm)	The distance of an object (mm)	DOF (mm)
1.0Vrms	439mm	464mm	450mm	25mm
2.5Vrms	285mm	315mm	300mm	30mm
4.0Vrms	130mm	172mm	150mm	42mm

Focal stack camera in all-in-focus imaging via an electrically tunable liquid crystal lens doped with multi-walled carbon nanotubes

Abstract

1. Introduction

2. Operating principle and sample preparation

2.1 Focal stack camera based on LC lens

1) DOF of the focal stack camera

2) Focal sweep speed of the focal stack camera

3) Algorithm of producing all-in-focus image

2.2 Sample preparation

3. Results and Discussions

4. Conclusions

Funding

Acknowledgments

References and links

Cited By

Figures (10)

Tables (2)

Equations (14)

Optics Express