Long working range light field microscope with fast scanning multifocal liquid crystal microlens array

Po-Yuan Hsieh; Ping-Yen Chou; Hsiu-An Lin; Chao-Yu Chu; Cheng-Ting Huang; Chun-Ho Chen; Zong Qin; Manuel Martinez Corral; Bahram Javidi; Yi-Pai Huang

doi:10.1364/OE.26.010981

1. Introduction

Based on the same structure as the conventional optical microscope, the light field microscope (LFM) was proposed [1–3]. In this microscope the only extra element is a microlens array (MLA) added at the image plane. As shown in Fig. 1 the sensor is displaced axially up to the lenslets focal plane. With this microscope the 4D spatial-angular information of the light field emitted by the specimen can be recorded on the sensor in just one single shot [4–9]. At the price of reducing the lateral resolution, the LFM has a collection of horizontal and vertical oblique views of the specimen with a large depth of field (DoF). Additionally, one can reconstruct the 3D structure of micro specimens through digital rendering processing [10–13].

Fig. 1 Schematic lay out of the LFM as reported by Levoy et al.

Download Full Size | PDF

Many proposals have been reported to enhance the spatial resolution of light field cameras. One method is to mechanically shift the intermediate image or microlens array by half microlens pitch for upper sampling [14–16], but the shaking issue and Nyquist theorem will limit the image quality. Furthermore, some computer vision researchers used the super-resolution [17–21] or hybrid camera method [22–24] to enhance the image resolution, but most methods need an all-in-focus 2D image as the database. Unfortunately, it is difficult to obtain an all-in-focus 2D image in the microscope system because of the narrow DoF of the optical microscope. In 2008, Georgiev et al. proposed the plenoptic 2.0 camera, which can increase the spatial resolution of the rendered image to the level of conventional cameras [25, 26]. However, the rendered image resolution will seriously decrease when objects are not at the optimal focus plane of the MLA. Therefore, the extension of the working range of the light field camera with acceptable resolution becomes an important issue for LFM applications based on the plenoptic 2.0 system.

In conventional optical microscope systems, the most straightforward solution to extend the working range is that axial-scanning the specimen by tuning the position or the effective focal length of the objective lens [27, 28]. However, it will induce non-uniform magnification and resolution issues in the LFM systems. To avoid those problems, one can use a spatially-modulated MLA and a deconvolution algorithm to extend the DoF of light field cameras [29], although it will sacrifice the depth resolution in the longitudinal direction. Or, one can use the bifocal MLA, which has two focal lengths of ordinary and extra-ordinary light, instead of the fixed MLA to double the working range of LFM [30, 31]. In addition, one can use the spatial multi-focus MLA, which is an array of interleaved micro-lenses with different focal lengths, to obtain even longer working range [32–34]. Unfortunately, the multi-focus MLA induces large blinds area between adjacent elemental images under the MLA because the effective microlens pitch becomes larger. Another solution is using electrically tunable devices, such as a liquid lens or liquid-crystal microlens array (LC-MLA), to adjust the effective focal length of the MLA [35, 36]. By time-multiplexed capturing a sequence of light field images with different focal length of LC-MLA, one can obtain a longer working range of the LFM. Compared with the bifocal MLA or the spatial multi-focus MLA, the LFM with the tunable multifocal LC-MLA has a dynamic working range and a smaller blind area between adjacent elemental images. However, the lens performance of the conventional multifocal LC-MLA is difficult to control, and the response time is too slow for the LFM.

In this paper, we proposed a high-resistance liquid crystal microlens array (HiR LC-MLA) for extending the working range of the LFM. Compared with prior LC-MLA, the proposed HiR LC-MLA has multifocal lengths with high quality point spread functions, low driving voltage and fast response time. By electrically adjusting the focal length of the HiR LC-MLA, we can change the optimal focus plane as if we replace the MLA with a proper numerical aperture ( $N A_{ML}$ ). Using multiple-steps focusing with fast scanning between different focal lengths of HiR LC-MLA, we can ensure that every object in the scene is clearly recorded on at least one light field image. Therefore, the total working range of the LFM is extended.

Additionally, in our proposed system, the intermediate image of the 3D specimen is placed at the virtual image space of HiR LC-MLA, in where the conditions of $N A_{ML}$ is much smaller, thus HiR LC-MLA can be fabricated with thin cell gap to achieve fast response time for time-multiplexed scanning. Compared with the confocal microscopes, which require a slow mechanical scanning process to record 3D objects, the proposed LFM with multifocal HiR LC-MLA can cover the larger working range with much less capturing time (3-5 frames only).

2. Light field theory

2.1 Light field microscope

The schematic diagram of the LFM system, as proposed by Levoy et al. [2, 3], is shown in Fig. 1. It can be considered a conventional telecentric-based microscope followed by a MLA placed just at the image plane. The sensor is displaced up to the lenslets focal plane, so that the gap $g = f_{ML}$ , being $f_{ML}$ the focal length of the lenslets. All along this paper, the conventional microscope in which the MLA is inserted will be named as the “host microscope”. Assuming a para-geometrical approach, each lenslet produces a microimage, which collects a bundle of rays passing through its center and having different slopes. All the spatial-angular information collected by the entire collection of microimages is grouped in the light field function $L (x', θ')$ . From this light field information it is easy to obtain the light field at the focal plane provided that we take into account the ABCD relation

(\begin{matrix} x \\ θ \end{matrix}) = (\begin{matrix} - f_{o b} / f_{L} & 0 \\ 0 & - f_{L} / f_{o b} \end{matrix}) (\begin{matrix} x' \\ θ' \end{matrix}) = (\begin{matrix} γ & 0 \\ 0 & M \end{matrix}) (\begin{matrix} x' \\ θ' \end{matrix}) .

In this equation

M = - f_{L} / f_{o b}

is the lateral magnification of the microscope and

γ = M^{- 1}

the angular magnification. It is also interesting that in telecentric microscopes the axial magnification,

β

, is calculated as

β = M^{2}

. Based in these relations we can state that

N A^{'} = N A / | M |

, where

N A = s i n α

and

N A^{'} = s i n α'

stand for the numerical aperture of the microscope objective (MO) as evaluated in the object space and the image space, respectively.

Much more interesting in terms of lateral resolution is the LFM design based in the plenoptic 2.0 concept [31] In this case the MLA and the sensor are displaced axially so that the intermediate image becomes the object of the light field camera part (see Fig. 2).

Fig. 2 The LFM working in the plenoptic 2.0 mode. In the scheme $p$ stands for the MLA pitch, and $δ$ for the pixel size.

Download Full Size | PDF

The light emitted by the object and therefore passing through the intermediate image is projected onto the image sensor by the MLA. Each microlens and its affiliated sensor area can be considered as a micro-camera, which will receive a portion of the rays from the intermediate image. Because the field of view (FOV) of each micro-camera is much smaller than that of the host microscope, every elemental image only records a smaller part of the specimen. Considering that the position of the intermediate image is at the optimal focus plane of the MLA, the gap, g, between the MLA and the sensor will be adjusted according to the lens law

\frac{1}{a} + \frac{1}{g} = \frac{1}{f_{M L}},

whereas the magnification between the intermediate image plane and the sensor plane is

M_{ML} = - g / a

. The angular resolution limit of this LFM is

χ = α / N

, where

N = | 2 α a / M p |

is the number microlenses covered by the beam of semi-angle

α

. In other words,

χ = | M p / 2 a |

.

In order to obtain the maximum profit of the pixels of the sensor, and therefore optimize the spatial and the angular resolution, the LFM should be setup in such a way that the adjacent microimages only touch but no overlap with one another. In Fig. 3, we show that setup for two LFM modes.

Fig. 3 The FOV of any microimage. (a) LFM working in real mode; (b) LFM working in virtual mode.

Download Full Size | PDF

2.2 Effective resolution of rendered images

In we consider a planar object placed at the focal plane of the LFM, the intermediate image appears just at the image plane of the host microscope and the final image at the sensor. In this case the resolution limit, as evaluated at the sensor plane is the result of the competence between two factors: the pixel size, $δ$ , and the size of the diffraction spot, $s_{λ} = λ g / ϕ_{ML}$ , where $ϕ_{ML}$ is the aperture diameter of the microlenses. In other words, the resolution limit at the sensor plane is given by

ρ'' = \max {δ, s_{λ}} .

However, since the LFM captures the images of 3D specimens it is convenient to evaluate the resolution also for parts of the specimen that are out from the focal plane. In this case, it is also necessary to evaluate the size of the defocused spot, as illustrated in Fig. 4.

Fig. 4 Scheme for the calculation of the size of the defocused light spot.

Download Full Size | PDF

It is straightforward to find that such size is given by

s (z) = ϕ_{M L} | \frac{g}{f_{M L}} - \frac{g}{a - z'} - 1 | = ϕ_{M L} | \frac{g}{f_{M L}} - \frac{g}{a - M^{2} z} - 1 |,

where

z

is the distance from the optimal focus plane to the out-of-focus plane (see Fig. 4) and

z^{'} = M^{2} z

. Naturally,

s (0) = 0

. Therefore, the resolution limit as evaluated at the sensor plane is

ρ'' (z) = \max {δ, s_{λ}, s (z)} .

From this value, we can evaluate the resolution limit at the intermediate plane as

ρ' (z) = \frac{ρ''}{| M_{M L} |} = \frac{| a - M^{2} z |}{g} ρ'',

and at the object plane of the LFM as

ρ (z) = \frac{ρ' (z)}{| M |} = \frac{f_{o b} | a - M^{2} z |}{f_{L} g} ρ'' (z) .

Following Ref [34]. we define the effective resolution ratio (ERR) as the quotient between the resolution limit provided by LFM and the one provided by the host microscope. Taking into account that the resolution limit of the host microscope, as evaluated in the image space, is

ρ'_{h s t} (z) = \max {δ, \frac{0.5 λ M}{N A}, 2 N A | z M |},

then, the ERR is defined for the LFM as

E R R (z) = \frac{ρ'_{h s t} (0)}{ρ' (z)} .

For this definition we have assumed that both the LFM and the host microscope are using sensors with the same pixel size, and that in the case of the host microscope the resolution limit is determined by the pixel size.

As an example we setup in our laboratory a LFM composed by an infinite-corrected 4x MO (Nikon S Fluor 4x, NA = 0.20) and a tube lens of focal length $f_{L} = 200 m m$ . The parameters of the MLA were: $f_{ML} = 1.8 m m$ , $ϕ_{ML} = 100 μ m$ , and $p = 110 μ m$ . The LFM was set in virtual mode such that $a = - 2.25 m m$ and $g = 1.0 m m$ . The pixel size was $δ = 3.7 μ m$ and the illumination wavelength $λ = 0.532 μ m$ . According to these parameters the number of perspectives recorded is $N = 2.05$ and the angular resolution limit $χ = 0.1 r a d$ . Next in Fig. 5 we have plotted the ERR corresponding to the LFM, together with the ERR corresponding to the host microscope, which is define as

Fig. 5 Effective resolution ratio corresponding to the host microscope and the LFM. The position $z = 0$ is at the optimal focus plane $a = - 2.25 m m$ .

Download Full Size | PDF

E R R_{h s t} (z) = \frac{ρ'_{h s t} (0)}{ρ'_{h s t} (z)} .

As shown in Fig. 5, the ERR curve changes slowly in the region near the optimal focus plane, when the intermediate image is within the DoF of the MLA. When the intermediate image is out from the DoF of the MLA, the ERR of the rendered image decreases extremely rapidly. The maximum ERR of the rendered image does not appear at the focal plane ( $z = 0$ ) due to the non-linear dependence of $M_{ML} (z)$ . Assuming, as it happens in this example, that $s_{λ} \geq δ$ , the LFM has the maximum ERR at the refocusing plane depth $| z | = λ a^{2} / M^{2} (ϕ_{M L}^{2} + λ | a |)$ , where $z$ and $a$ have the same sign. The maximum ERR can be described as

E R R_{\max} = \frac{δ \cdot ϕ_{M L}}{λ \cdot | a |} + \frac{δ}{ϕ_{M L}},

where

a

is the distance from the optimal focus plane of the MLA. In other words, the MLA is set as the original plane (

a = 0

).

2.3 Working range

According to Eq. (9), we can define the working range of the LFM. The objects in the working range are clearly recorded on elemental images and reconstructed with acceptable resolution. If we assume that the resolution limitation $ρ' (z)$ is determined by the defocus effect but not the diffraction effect, the working range of the LFM can be described as

W R = \frac{1}{M^{2}} \cdot \frac{2 δ | a |}{E R R_{\min} \cdot ϕ_{M L}},

where

E R R_{m i n}

is the minimum acceptable resolution ratio of observer [42]. A comparison of Eqs. (11) and (12) shows that the working range (WR) and effective resolution ratio (ERR) are trade-off. If we design a LFM with the highest

E R R_{m a x}

, the working range will shrink and become incapable of capturing thick 3D specimens.

To extend the working range of the LFM, and to maintain the acceptable effective resolution ratio at the same time, we used the multifocal LC-MLA instead of the fixed MLA in the LFM system as shown in Fig. 6. Assuming that the gap between the LC-MLA and the sensor is constant, once we change the focal length of LC-MLA, the working range will shift along the optical axis synchronously. By combining the light field images with different focal lengths of the LC-MLA, the total working range of the LFM will be much extended as Fig. 7(b). Furthermore, to generate a smooth variation of ERR value at different depth planes, the turning points of each ERR curve should be connected to the gradual region of adjacent ERR curves. Hence, the numerical apertures of LC-MLA ( $N A_{ML}$ ) are designed as $N A_{n + 1} \approx N A_{n} + λ / ϕ_{M L}$ . The extended working range can be described as

W R_{t o t a l} = \frac{1}{M^{2}} (| a_{N} - a_{1} | + \frac{δ | a_{N} + a_{1} |}{ϕ_{M L} \cdot E R R_{\min}}),

where

a_{N}

is the optimal focus plane with the largest

N A_{M L}

.

Fig. 6 The extended working range of the LFM with multifocal LC-MLA.

Download Full Size | PDF

Fig. 7 Effective resolution ratio (ERR) and working range of the LFM (a) with the fixed MLA and (b) with the multifocal LC-MLA. The MLA plane is set at $a = 0$ . The symbol $a$ is the depth from the intermediate image plane to the MLA plane.

Download Full Size | PDF

3. High-resistance LC microlens array (HiR LC-MLA)

As we mentioned, the multifocal LC-MLA has electrically tunable focal length by applying the driving voltages on it. However, it is difficult to modify the ideal parabolic lens profile at every focal plane with a single adjustment parameter. In addition, the response time of conventional LC-MLA is too slow for the LFM. In this paper, we proposed the HiR LC-MLA, which has a thin layer of high-resistance material coated on the electrode patterns. The HiR LC-MLA is controllable with not only the driving voltage, but also the frequency of the driving signal. The lens performance of HiR LC-MLA can be optimized by adjust the composition of driving voltage and frequency. Moreover, the HiR LC-MLA has advantages such as lower driving voltage and faster response time for the time-multiplexed scanning between different focal lengths.

3.1 Multifocal HiR LC-MLA

As Fig. 8 shows, the high-resistance liquid crystal microlens array (HiR LC-MLA) consists of two planar electrode-glass substrates separated by a homogeneous gap as the LC cell. The electrode pattern on the top substrate is an array of hexagonal micro-holes, on which a thin layer of $N b_{2} O_{5}$ was coated as the high-resistance (HiR) layer. The bottom substrate has a transparent planar ITO electrode. Between them, the nematic LC material was injected into the LC cell. As shown in Fig. 9, the constitution of the HiR layer and LC layer can be considered an R-C circuit [37–39]. When an alternating current (AC) voltage signal is applied to the electrodes, it will induce a gradient voltage distribution on the nodes of the R-C circuit. The potential difference will generate a parabolic lens-like phase retardation in the LC cell as shown in Fig. 10. Therefore, the focal length of the HiR LC-MLA is electrically tunable by adjusting the driving voltage and frequency [37]. A higher driving frequency will induce a larger phase retardation difference between the edge and center region of each LC microlens; simultaneously, the focal length of the HiR LC-MLA will shorten. The focal length of the LC microlens can be described as

f_{M L} = \frac{ϕ_{M L}^{2}}{8 \cdot Δ n \cdot d_{L C}},

where

ϕ_{M L}

is the microlens aperture size,

d_{L C}

is the thickness of the LC layer, and

Δ n

is the refractive index difference between the center and the edge of the microlens aperture [40]. As shown in Fig. 10, the lens effect in the conventional LC-MLA was generated by the fringe field emitted from the edge electrodes; hence, the orientation of the LC molecules in the center of the microlens aperture was difficult to control. However, the HiR LC-MLA has a smooth electric field and will induce symmetrical lens profile. As shown in Fig. 11, the interference patterns of the conventional LC-MLA and HiR LC-MLA show that the HiR LC-MLA had less disclination line effect than the conventional LC-MLA. The driving voltage of the HiR LC-MLA was half or less than half of the conventional LC-MLA with the identical lens power.

Fig. 8 (a) Section view and (b) hexagonal electrode pattern of the HiR LC-MLA.

Download Full Size | PDF

Fig. 9 R-C circuit model of an LC microlens with the HiR layer.

Download Full Size | PDF

Fig. 10 Electric field distribution and LC molecule orientation of the (a) conventional fringe-field-controlled LC lens without the HiR layer and (b) HiR LC lens.

Download Full Size | PDF

Fig. 11 Interference patterns of the (a) conventional fringe-field-controlled LC-MLA and (b) HiR LC-MLA.

Download Full Size | PDF

Figure 12 shows the interference patterns (IPs) and point spread functions (PSFs) of the HiR LC-MLA with the driving signal. The aperture size of each microlens was $350 μm$ , and the LC cell gap was $60 μm$ . As we expected, the IPs show the parabolic lens-like phase retardation distribution of HiR LC-MLA at different focal lengths. The relation of the HiR LC-MLA focal length and driving signal is shown in Fig. 13. By applying a voltage of 2.6 $V_{r m s}$ and changing the frequency from $80 kHz$ to $150 kHz$ , we decreased the focal length of the HiR LC-MLA from $3.1 mm$ to $1.7 mm$ ; equivalently, the numerical aperture ( ${NA}_{ML}$ ) value of the HiR LC-MLA changed from $N A_{M L} = 0.056$ to $N A_{M L} = 0.103$ . Compared with the fixed MLA, the HiR LC-MLA has tunable lens power; but it also induces the unsymmetrical side lobes in the PSFs especially at the short focal length case as shown in Fig. 12. The non-uniform high-order aberration on the light field images should be considered in the image enhancement processing.

Fig. 12 Interference patterns (IPs) and point spread functions (PSFs) of the HiR LC-MLA with different focal lengths. Microlens aperture size: $ϕ_{ML} = 350 μ m$ ; LC cell gap: $d_{L C} = 60 μ m$ .

Download Full Size | PDF

Fig. 13 Focal length of the HiR LC-MLA with different driving frequencies. Microlens aperture size: $ϕ_{M L} = 350 μ m$ ; LC cell gap: $d_{L C} = 60 μ m$ ; driving voltage: $V = 2.6 V_{rms}$ .

Download Full Size | PDF

3.2 Virtual mode for fast scanning

As shown in Fig. 7(b), the proposed LFM with fast scanning multifocal LC-MLA should have a larger working range and acceptable resolution compared to the conventional LFM with the fixed MLA. Therefore, two boundary conditions should be satisfied:

1. The effective resolution ratio $E R R \geq E R R_{m i n}$ in the entire working range. $\frac{δ \cdot ϕ_{M L}}{λ \cdot | a |} + \frac{δ}{ϕ_{M L}} \geq E R R_{\min} .$
2. The working range of the LFM should be equal to or larger than that of the host microscope as $W R \geq d_{h s t}$ $\frac{1}{M^{2}} \cdot \frac{2 δ | a |}{E R R_{\min} \cdot ϕ_{M L}} \geq d_{h s t} .$

where

d_{h s t}

is the working range of the host microscope, which is equal to its DoF [41].

The intermediate image of the specimen, which is considered the object of the light field camera part, is commonly placed in front of the MLA ( $a > 0$ ) as shown in Fig. 14(a) [42]. In this case, a higher optical power of the MLA is required to converge the light onto the image sensor. Therefore, we proposed to place the intermediate image in the virtual image space of the MLA ( $a < 0$ ) as shown in Fig. 14(b) because the length of the entire optical system is shorter, and the condition of $N A_{M L}$ is much smaller than those in the real image case. According to Eqs. (15) and (16), if the intermediate image is placed in the real mode ( $a > 0$ ) as shown in Fig. 14(a), we have the following limits of $N A_{M L}$ ,

\frac{λ \cdot (E R R_{\min} - \frac{δ}{ϕ_{M L}})}{2 δ} + \frac{α'}{2 κ} \leq N A_{M L} \leq \frac{δ}{M^{2} \cdot d_{h s t} \cdot E R R_{\min}} + \frac{α'}{2 κ},

where

κ = p / ϕ_{M L}

; in contrast, if the intermediate image is placed in the virtual mode (

a < 0

) as shown in Fig. 14(b), the limitation of

N A_{M L}

is

\frac{α'}{2 κ} - \frac{δ}{M^{2} \cdot d_{h s t} \cdot E R R_{\min}} \leq N A_{M L} \leq \frac{α'}{2 κ} - \frac{λ \cdot (E R R_{\min} - \frac{δ}{ϕ_{M L}})}{2 δ} .

The condition of

N A_{M L}

for the virtual mode is one order smaller than that for the real mode. Therefore, the LC cell gap of HiR LC-MLA can be shrunk, and the response time is considerably shorter.

Fig. 14 Intermediate image placed in the (a) real image space and (b) virtual image space of LC-MLA. The thickness and numerical aperture of the LC-MLA in real mode are larger than those in virtual mode.

Download Full Size | PDF

In the conventional fringe-field-controlled LC-MLA as Fig. 10(a), the aspect ratio of the aperture size and LC cell gap was designed as $ϕ_{M L} / d_{L C} = 5 ~ 6$ for acceptable lens quality [43]. However, the LC molecules orientation of the HiR LC-MLA was electrically tunable by modulating the driving voltage and frequency [37]. With the HiR LC-MLA, we could achieve the aspect ratio of $ϕ_{M L} / d_{L C} = 10$ , where the LC lens with small cell gap and fast response time becomes possible (Table 1).

Table 1. HiR LC-MLA with a large aperture size, high $N A_{M L}$ , and fast response time.

View Table

In this paper, we designed three types of HiR LC-MLA with different aperture sizes, cell gaps, and LC materials as the multifocal LC-MLA. All types were coated with 20 nm ${Nb}_{2} O_{5}$ as the high-resistance layer. The first structure ( $ϕ_{M L} = 350 μ m$ , $d_{L C} = 60 μ m$ , $Δ n = 0.21$ ) has large aperture size and LC cell gap, which provided large $N A_{M L} = 0.1$ but slow response time (~15 seconds). To reduce the response time of the HiR LC-MLA, we shortened the LC cell gap from 60 μm to 15 μm and shrank the aperture size from 350 μm to 100 μm. Using high-birefringence LC materials, the numerical aperture of the second structure ( $ϕ_{M L} = 100 μ m$ , $d_{L C} = 15 μ m$ , $Δ n = 0.29$ ) was increased to $N A_{M L} = 0.15$ , and the response time was reduced to 0.12 seconds. In the third structure, we shortened the LC cell gap to 10 μm; hence, the response time of the third structure ( $ϕ_{M L} = 100 μ m$ , $d_{L C} = 10 μ m$ , $Δn = 0.21$ ) was reduced to 0.08 seconds. The video showing the response time of fast response HiR LC-MLA is shown in https://youtu.be/kVy64Uj017A.

4. Experiments and results

4.1 Experimental setup

In this paper, the fast response HiR LC-MLA with a small aperture size $ϕ_{M L} = 100 μ m$ and a small LC cell gap $d_{L C} = 10 μ m$ was used. The intermediate image of the specimen was placed in the virtual image space of HiR LC-MLA ( $a < 0$ ). The experimental setup of the long working range LFM with the multifocal HiR LC-MLA is shown in Fig. 6. The infinity-corrected 4x MO (Nikon S Fluor 4x/0.20) was used. The focal length of the tube lens is $f_{L} = 200 m m$ , and the magnification of microscopic system is $| M | = 4$ . Following the tube lens were the linear polarizer and HiR LC-MLA; the polarization direction of the linear polarizer was parallel to the alignment direction of the HiR LC-MLA. Then, the images sensor (pixel size $δ = 3.7 μ m$ ) was effectively placed at $g = 1 m m$ behind the HiR LC-MLA. By adjusting five focal lengths of the HiR LC-MLA as $f_{M L} = 1.2 m m, 1.4 m m, 1.8 m m, 2.4 m m, 3.5 m m$ ( $N A_{M L} = 0.0143 ~ 0.0417$ ), we would capture five light field images with the corresponding focal lengths. A wing of a drain fly was placed at the focal plane of the objective lens as the specimen. The LC-MLA plane was set at the original plane ( $a = 0$ ). The intermediate image of the fly’s wing tip was near the LC-MLA plane ( $a = - 1.4 m m$ ), and the root of the wing was far from it ( $a = - 6.0 m m$ ).

4.2 Rendered images

The raw light field image with $f_{M L} = 3. 5 m m$ is shown in Fig. 15; every light field image has $188 \times 150$ elemental images stacked as a hexagonal array. The resolution of each elemental image is $31 \times 30$ . Figure 15 also shows the partial enlarged views of the raw light field image with $f_{M L} = 3.5 m m$ , in which the optimal focus plane of the HiR LC-MLA is at $a = - 1.4 m m$ . Hence, the elemental images of the tip of the wing are clear (in-focus); in contrast, the elemental images of the root of the wing are blurry because the objects are out-of-focus. Figure 16 shows the rendered images, which refocused at different depth planes. The top row images were rendered from the single light field image captured by a fixed MLA with $f_{M L} = 3.5 m m$ . The bottom row images were rendered from the light field image stack captured with the multifocal HiR LC-MLA. As Fig. 16 shows, when refocusing at the tip of the drain fly’s wing ( $a = - 1.4 m m$ ), which was near the LC-MLA plane, both rendered images from the single light field image and light field image stack could provide rich and colorful details of the veins. However, when refocusing at the root of the wing ( $a = - 6.0 m m$ ), only the light field image stack could provide a rendered image with acceptable resolution. The single light field image with $f_{M L} = 3.5 m m$ , which had a shallower working range, could not reconstruct clear images of the object far from the LC-MLA plane. Therefore, the root of the wing in top row image was blurry even when refocusing at the correct depth plane.

Fig. 15 Raw light field image with $f_{M L} = 3.5 m m$ . The partial enlarged views show that (a) the root ( $a = - 6.0 m m$ ) of the wing is out-of-focus, and (b) the tip $(a = - 1.4 m m)$ of the wing is in-focus.

Download Full Size | PDF

Fig. 16 Rendered images of the LFM with (a) fixed focal length MLA and (b) multifocal HiR LC-MLA that refocuses at different depth planes.

Download Full Size | PDF

Figure 17 shows the effective resolution ratio (ERR) curves of the light field images with different $N A_{M L}$ in the experiment. The light field image with small $N A_{M L}$ will have a higher ERR but a shorter working range; in contrast, the LC-MLA with high $N A_{M L}$ will induce a lower ERR but a longer working range of the LFM. In this paper, we set the minimum acceptable effective resolution ratio $E R R_{m i n} = 0.087$ for human eye condition [44]. As shown in Fig. 17, the total working range is from $a = - 1 m m$ to $a = - 7.99 m m$ in the intermediate space. In other words, the working range in the specimen space was enlarged from 62.5 μm to 436.8 μm, which is 7 times larger than that of the conventional LFM with a fixed MLA. Compared with the confocal microscopes, which must stack hundreds of slice image to generate the 3D depth structure of a specimen, the proposed LFM with HiR LC-MLA can record and reconstruct the same depth range with only 3 ~5 light field images.

Fig. 17 Effective resolution ratio and total working range of the light field microscope with the tunable-focus HiR LC-MLA, which covers from $a = - 1 m m$ to $a = - 7.99 m m$ in the intermediate space.

Download Full Size | PDF

4.3 Oblique view results

Except to render images that refocus at different depth planes, the LFM can also provide rendered images with oblique view information. The video of oblique view images is shown in https://youtu.be/xLv4W8re53E; the viewing angle is from $19.29 °$ to $- 19.29 °$ . The viewing angle was determined by the size of the HiR LC-MLA and image sensor but not the microlens pitch or numerical aperture $N A_{M L}$ . However, with a smaller microlens pitch, both angular resolution and depth resolution of the LFM will increase [34].

5. Conclusions

The light field microscope can record the 3D information of the specimen in a single shoot without mechanical scanning. However, the working range still limit the applications of light field microscope. Therefore, we have proposed a light field microscope with high-resistance liquid crystal microlens array (HiR LC-MLA) to much extend the working range. The HiR LC-MLA, which is coated with a high resistance layer of $N b_{2} O_{5}$ , has more symmetric lens profile and lower driving voltage than the conventional LC-MLA. By adjusting the focal length of the HiR LC-MLA and capturing a consequence of light field images, we can record and reconstruct the 3D specimen with a continuous and acceptable effective resolution ratio (ERR) in a large depth range.

In the proposed experiment system, the intermediate image of specimen was placed in the virtual mode ( $a < 0$ ), thus the fast response HiR LC-MLA ( $R s p . = 0.08 s e c o n d .$ ) can be used. By adjusting five appropriate focal lengths of the HiR LC-MLA as $f_{M L} = 1.2 ~ 3.5 m m$ ( $N A_{M L} = 0.0143 ~ 0.0417$ ) to record the light field image stack, the working range of the proposed light field microscope was much extended from 62.5 μm to 436.8 μm (Almost 7 times). With the long working range, it can capture thick specimens or microorganism spreading in large depth range. However; the jelly effect may appear in the depth boundary of two light field images when observing the dynamic specimens. For real time capturing the light field images, the response time of the HiR LC-MLA and the image sensor should be shorten by at least one order.

Funding

National Chiao Tung University (NCTU) in Taiwan; Ministry of Science and Technology (MOST) in Taiwan (contract No. NSC101-2221-E-009-120-MY3); Spanish Ministry of Economy and Competitiveness in Spain (grant No. DPI2015-66458-C2-1-R); Generalitat Valenciana in Spain (project No. PROMETEOII/2014/072).

Acknowledgments

The polyimide was generously provided by the Industrial Technology Research Institute (ITRI), Taiwan.

References and links

1. J.-S. Jang and B. Javidi, “Three-dimensional integral imaging of micro-objects,” Opt. Lett. 29(11), 1230–1232 (2004). [CrossRef] [PubMed]

2. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006). [CrossRef]

3. M. Levoy, Z. Zhang, and I. McDowall, “Recording and controlling the 4D light field in a microscope using microlens arrays,” J. Microsc. 235(2), 144–162 (2009). [CrossRef] [PubMed]

4. G. Lippmann, “Epreuves reversibles donnant la sensation du relief,” J. Phys. Theor. Appl. 7(1), 821–825 (1908). [CrossRef]

5. E. H. Adelson and J. Y. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. 14(2), 99–106 (1992). [CrossRef]

6. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Stanford Tech. Rep. CTSR (Stanford University, 2005), pp. 1–11.

7. R. Ng, “Digital light field photography,” Ph.D, Department of Computer Science (Stanford University California, 2006).

8. C. Zhou and S. K. Nayar, “Computational cameras: convergence of optics and processing,” IEEE Trans. Image Process. 20(12), 3322–3340 (2011). [CrossRef] [PubMed]

9. E. Y. Lam, “Computational photography with plenoptic camera and light field capture: tutorial,” J. Opt. Soc. Am. A 32(11), 2021–2032 (2015). [CrossRef] [PubMed]

10. E. H. Adelson and J. R. Bergen, “The plenoptic function and the elements of early vision,” in Computational Models of Visual Processing (MIT Press, 1991), pp. 3–20.

11. M. Levoy and P. Hanrahan, “Light field rendering,” in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (ACM1996), pp. 31–42.

12. A. Isaksen, L. McMillan, and S. J. Gortler, “Dynamically reparameterized light fields,” in Proceedings of the 27th annual conference on Computer graphics and interactive techniques (ACM Press/Addison-Wesley Publishing Co.2000), pp. 297–306.

13. D.-H. Shin and E.-S. Kim, “Computational integral imaging reconstruction of 3D object using a depth conversion technique,” J. Opt. Soc. Korea 12(3), 131–135 (2008). [CrossRef]

14. L. Erdmann and K. J. Gabriel, “High-resolution digital integral photography by use of a scanning microlens array,” Appl. Opt. 40(31), 5592–5599 (2001). [CrossRef] [PubMed]

15. P. Didyk, E. Eisemann, T. Ritschel, K. Myszkowski, and H.-P. Seidel, “Apparent display resolution enhancement for moving images,” ACM Trans. Graph. 29(4), 113 (2010). [CrossRef]

16. T.-H. Jen, X. Shen, G. Yao, Y.-P. Huang, H.-P. D. Shieh, and B. Javidi, “Dynamic integral imaging display with electrically moving array lenslet technique using liquid crystal lens,” Opt. Express 23(14), 18415–18421 (2015). [CrossRef] [PubMed]

17. T. E. Bishop, S. Zanetti, and P. Favaro, “Light field superresolution,” in Proceedings IEEE Conference on Computational Photography (IEEE, 2009), pp. 1–9.

18. Y. Wang, G. Hou, Z. Sun, Z. Wang, and T. Tan, “A simple and robust super resolution method for light field images,” in Proceedings IEEE Conference on Image Processing (IEEE, 2016), pp. 1459–1463. [CrossRef]

19. H. Li, C. Guo, and S. Jia, “High-resolution light-field microscopy,” in Frontiers in Optics (Optical Society of America, 2017), paper FW6D.3.

20. M. Rossi and P. Frossard, “Light Field Super-Resolution Via Graph-Based Regularization,” arXiv preprint arXiv:1701.02141 (2017).

21. C.-H. Lu, S. Muenzel, and J. Fleischer, “High-Resolution Light-Field Microscopy,” in Computational Optical Sensing and Imaging on Microscopy and Tomography I, OSA Technical Digest (Optical Society of America, 2013), paper CTh3B.2.

22. V. Boominathan, K. Mitra, and A. Veeraraghavan, “Improving resolution and depth-of-field of light field cameras using a hybrid imaging system,” in Proceedings IEEE Conference on Computational Photography (IEEE, 2014), pp. 1–10. [CrossRef]

23. J. Wu, H. Wang, X. Wang, and Y. Zhang, “A novel light field super-resolution framework based on hybrid imaging system,” in Proceedings IEEE Conference on Visual Communications and Image Processing (IEEE, 2015), pp. 1–4. [CrossRef]

24. M. Z. Alam and B. K. Gunturk, “Hybrid Light Field Imaging for Improved Spatial Resolution and Depth Range,” arXiv preprint arXiv:1611.05008 (2016).

25. A. Lumsdaine and T. Georgiev, “Full resolution lightfield rendering,” Indiana University and Adobe Systems, Tech. Rep (2008).

26. T. Georgiev and A. Lumsdaine, “Focused plenoptic camera and rendering,” J. Electron. Imaging 19(2), 021106 (2010). [CrossRef]

27. C.-W. Chen, M. Cho, Y.-P. Huang, and B. Javidi, “Three-dimensional imaging with axially distributed sensing using electronically controlled liquid crystal lens,” Opt. Lett. 37(19), 4125–4127 (2012). [CrossRef] [PubMed]

28. M. Martı, P.-Y. Hsieh, A. Doblas, E. Sánchez-Ortiga, G. Saavedra, and Y.-P. Huang, “Fast axial-scanning widefield microscopy with constant magnification and resolution,” J. Disp. Technol. 11(11), 913–920 (2015). [CrossRef]

29. M. Martínez-Corral, B. Javidi, R. Martínez-Cuenca, and G. Saavedra, “Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays,” Appl. Opt. 43(31), 5806–5813 (2004). [CrossRef] [PubMed]

30. K.-C. Kwon, M.-U. Erdenebat, Y.-T. Lim, K.-I. Joo, M.-K. Park, H. Park, J.-R. Jeong, H.-R. Kim, and N. Kim, “Enhancement of the depth-of-field of integral imaging microscope by using switchable bifocal liquid-crystalline polymer micro lens array,” Opt. Express 25(24), 30503–30512 (2017). [CrossRef] [PubMed]

31. K.-C. Kwon, Y.-T. Lim, C.-W. Shin, M.-U. Erdenebatz, J.-M. Hwang, and N. Kim, “Enhanced depth-of-field of an integral imaging microscope using a bifocal holographic optical element-micro lens array,” Opt. Lett. 42(16), 3209–3212 (2017). [CrossRef] [PubMed]

32. J.-S. Jang and B. Javidi, “Large depth-of-focus time-multiplexed three-dimensional integral imaging by use of lenslets with nonuniform focal lengths and aperture sizes,” Opt. Lett. 28(20), 1924–1926 (2003). [CrossRef] [PubMed]

33. T. Georgiev and A. Lumsdaine, “The multifocus plenoptic camera,” Proc. SPIE 8299, 829908 (2012). [CrossRef]

34. C. Perwass and L. Wietzke, “Single lens 3D-camera with extended depth-of-field,” Proc. SPIE 8291, 829108 (2012). [CrossRef]

35. A. Tolosa, R. Martínez-Cuenca, A. Pons, G. Saavedra, M. Martínez-Corral, and B. Javidi, “Optical implementation of micro-zoom arrays for parallel focusing in integral imaging,” J. Opt. Soc. Am. A 27(3), 495–500 (2010). [CrossRef] [PubMed]

36. Y. Lei, Q. Tong, X. Zhang, H. Sang, A. Ji, and C. Xie, “An electrically tunable plenoptic camera using a liquid crystal microlens array,” Rev. Sci. Instrum. 86(5), 053101 (2015). [CrossRef] [PubMed]

37. A. F. Naumov, M. Yu. Loktev, I. R. Guralnik, and G. Vdovin, “Liquid-crystal adaptive lenses with modal control,” Opt. Lett. 23(13), 992–994 (1998). [CrossRef] [PubMed]

38. Y.-C. Chang, T.-H. Jen, C.-H. Ting, and Y.-P. Huang, “High-resistance liquid-crystal lens array for rotatable 2D/3D autostereoscopic display,” Opt. Express 22(3), 2714–2724 (2014). [CrossRef] [PubMed]

39. J. Beeckman, I. Nys, O. Willekens, and K. Neyts, “Optimization of liquid crystal devices based on weakly conductive layers for lensing and beam steering,” J. Appl. Phys. 121(2), 023106 (2017). [CrossRef]

40. Y. P. Huang, L. Y. Liao, and C. W. Chen, “2‐D/3‐D switchable autostereoscopic display with multi‐electrically driven liquid‐crystal (MeD‐LC) lenses,” J. Soc. Inf. Disp. 18(9), 642–646 (2010). [CrossRef]

41. Y.-T. Lim, J.-H. Park, K.-C. Kwon, and N. Kim, “Analysis on enhanced depth of field for integral imaging microscope,” Opt. Express 20(21), 23480–23488 (2012). [CrossRef] [PubMed]

42. J.-S. Jang, F. Jin, and B. Javidi, “Three-dimensional integral imaging with large depth of focus by use of real and virtual image fields,” Opt. Lett. 28(16), 1421–1423 (2003). [CrossRef] [PubMed]

43. S. Oka, T. Naganuma, T. Koito, Y. Yang, and S. Komura, “15.5: Invited Paper: High Performance Autostereoscopic 2D/3D Switchable Display Using Liquid Crystal Lens,” in SID Symposium Digest of Technical Papers (Wiley Online Library, 2013), pp. 150–153. [CrossRef]

44. H.-A. Lin, “Response Time Acceleration and Image Quality Analysis of Light Field Microscope with Liquid Crystal Lens Array,” in Institute of Lighting and Energy Photonics (National Chiao Tung University, 2017).

	Large Aperture	High NA	Fast Response
Aperture (D)	350 μm	100 μm	100 μm
Cell gap $(d_{L C})$	60 μm	15 μm	10 μm
$D / d_{L C}$	5.8	6.67	10
LC birefringence $(Δ n)$	0.21	0.29	0.21
*$f_{M L}$*	1.7 mm – 3.1 mm	0.33 mm – 0.55 mm	1.2 mm – 3.5 mm
*$N A_{M L}$*	0.10 – 0.06	0.15 – 0.09	0.042 – 0.014
Interference pattern
Response time	> 1 sec.	0.12 sec.	0.08 sec.

Long working range light field microscope with fast scanning multifocal liquid crystal microlens array

Abstract

1. Introduction

2. Light field theory

2.1 Light field microscope

2.2 Effective resolution of rendered images

2.3 Working range

3. High-resistance LC microlens array (HiR LC-MLA)

3.1 Multifocal HiR LC-MLA

3.2 Virtual mode for fast scanning

4. Experiments and results

4.1 Experimental setup

4.2 Rendered images

4.3 Oblique view results

5. Conclusions

Funding

Acknowledgments

References and links

Cited By

Figures (17)

Tables (1)

Equations (18)

Optics Express