Open Access
Add to BibSonomyAbstract
A reconfigurable telecentric imaging system that can simultaneously capture multispectral and multi-polarization information in a single snapshot is demonstrated. The proposed design utilizes a telecentric imaging objective and a light pipe for image multiplexing. An array of filters is used to filter each of the multiplexed images at an intermediate image plane. In this paper, we will discuss the system configuration and present the experimental results.
© 2014 Optical Society of America
1. Introduction
Multispectral and polarization imaging is a growing tool in remote sensing, material analysis, and biomedical imaging [1–4]. There are a number of methods for image multiplexing which include the use of a beamsplitter, lens array [5], hologram [6], Wollaston prism [7], image slicer [8], and a light pipe to generate multiple images [9]. A key challenge in developing a multimodal spectral and polarimetric imaging system is capturing multispectral and multi-polarization images simultaneously. Most current snapshot multispectral imaging devices cannot simultaneously measure polarization and spectral information. There are a few instances of snapshot spectropolarimeters; these are computationally intensive and they are not reconfigurable [10–12]. The imaging system described in this paper uses a low-cost light pipe and telecentric imaging objective to multiply images and measure spectral and polarization information in a single snapshot without the need for computationally intensive reconstruction.
2. System concept
The snapshot multispectral and polarization imaging system creates an identical array of images and then filters each of these images simultaneously using an array of optical filters. A diagram demonstrating this concept is shown in Fig. 1. As an example, assume an imaging objective forms an image that contains the entire visible spectrum at the input end of a light pipe and the image is horizontally polarized. This image is multiplied by the light pipe and all of the multiplied images are focused by a relay lens onto an intermediate image plane. A filter array containing three linear polarizers rotated at 0, 45, 90, a left circular analyzer and 5 spectral filters is placed at the intermediate image plane and filters each sub-image. All filtered sub-images are then imaged onto a monochrome sensor. Therefore, in a single snapshot, multispectral and multiple polarization images can be captured simultaneously.
The multispectral and polarization imaging system can be broken up into five distinct sub-systems: an imaging objective, an image multiplexing light pipe, a relay system, a filter array, and an imaging lens.
2.1 Light pipe: design considerations
The key element in this design is the image multiplexing light pipe. Each reflection in the light pipe creates a virtual object outside the light pipe, which creates an identical copy of the original object placed at the input end of the light pipe. The number of reflections, n, for an on-axis object is given by:
where is the floor operation which rounds down to the nearest integer, is the half cone angle of the incident light, h is the height of the light pipe, and L is the length of the light pipe. The number of images, N, created is 2n + 1. The dimensions of the light pipe are optimized to constrain the number of reflections of the incident light. The path taken for each virtual object through a light pipe optimized for 5 images can be seen in Fig. 2(a). The path taken for each virtual object is colored red, green, and blue for 0, 1, and 2 reflections in the light pipe. As shown in the spot diagram in Fig. 2(a) where a perfect paraxial lens is used to focus the virtual objects, the light pipe will not introduce aberrations if the reflective surface is polished flat.
Fig. 2 Different configurations for image multiplexing using the light pipe. The telecentric imaging objective is the best configuration, since it most closely resembles an object placed at the input end of the light pipe.
The object placed at the input end of the light pipe can be a physical object, or it can be the image from another optical system. An example of an image formed at the input end of the light pipe using a non-telecentric imaging objective is shown in Fig. 2(b). Typically, the chief ray of the image is divergent. There are two potential issues: (1) the flux in each sub-image is not the same, especially for the edges of the sub-image, and (2) the light will leak into other sub-images. The numerical aperture of each sub-image is significantly different. Ideally, one-third of the rays from the top object (blue rays), should be focused to the top sub-image, I1. Figure 2(b) shows very few rays are focused to the desired sub-image I1, instead most of those rays are focused into the sub-image above I1. Therefore, this configuration is not desirable.
In order to obtain a cone angle of light large enough for the light pipe and to prevent parallax between the sub-images, a diffuser was typically placed at the input end of the light pipe [9]. Since the diffuser has a finite thickness, one point in the object was modelled in Fig. 2(c) as a series of point sources along the entire thickness of the diffuser. From the spot diagram in Fig. 2(c), it can be seen that the finite thickness of the diffuser causes significant blurring. The situation is even worse in reality, since the diffuser also scatters light radially. Additionally, light scattered inside the diffuser causes loss of light. In addition to the reduced spatial resolution and the introduction of light loss, the diffuser in the intermediate image plane also causes any feature on the diffuser to be relayed to the detector.
Here we propose a light efficient solution which uses a telecentric imaging objective to avoid the image degradation and cross-talk. An example of a light pipe with a telecentric image formed at the input end of the light pipe can be seen in Fig. 2(d). The bounds of each sub-image is indicated by the arrows. The cone angle and dimensions for the light pipe were optimized for one reflection from the top surface and bottom surface for the on-axis point source. Since the off-axis points are closer to the top or bottom surface of the light pipe, some rays closest to these surfaces may reflect off the top and bottom surface, causing those rays to reach the area outside the desired 3 sub-images. For the telecentric configuration, the amount of light focused outside the desired 3 sub-images is reduced.
A telecentric imaging objective most closely resembles an object placed at the input end of the light pipe. It increases the light into the desired sub-images and there is no image degradation due to scattering inside of a diffuser. The image can be replicated N times by the light pipe as long as the telecentric image has the necessary cone angle for the light pipe. By changing the cone angle of incident light or the dimensions of the light pipe, the number of multiplied images can easily be scaled from 3x3 to 5x5 or even larger. The aspect ratio of the light pipe is typically the same as the aspect ratio of the detector in order to fully utilize all pixels in the sensor.
2.2 Telecentric imaging objective
An essential requirement of the imaging objective is the telecentricity in image space since a telecentric image focused at the input end of the light pipe is the best configuration to achieve identical sub-images for multispectral and multi-polarization imaging. Once the dimensions of the light pipe are finalized, any imaging objective that is image-space telecentric and has a cone angle of can be used in this system. Therefore, an imaging objective optimized for microscopy could be swapped with an imaging objective optimized for wide FOV imaging without any modification to the light pipe, relay system, or imaging lens.
This system can be easily adapted to multiple applications by replacing the telecentric imaging objective without any other modifications to the system and it can simultaneously measure both multispectral and polarization information. If a filter array is used that consists of only narrowband spectral filters, then an approximation of the spectrum consisting N2 points can be constructed. Alternatively, if polarization elements are simultaneously used, then Stokes vectors and an approximation of the spectrum can be computed simultaneously. Both the spectral and polarization information can be analyzed in a large number of applications, such as cancer detection, microscopy, endoscopy, and image guided surgery to name a few.
2.3 Relay system and filter array
The relay lens images the identical sub-images created by the light pipe onto an intermediate image plane creating an NxN array of identical images. For an optimal relay system, the field of view should be large to cover NxN sub-images and the aberrations should be well-controlled so that the sub-images can be registered with minimal image processing. In addition, telecentricity in image space is advantageous because the polarizers and the filters are sensitive to incident angles. Telecentricity will ensure each point has the same performance.
An array of filters is placed in this intermediate image plane to individually filter each of the N2 identical images. This filter array can consist of spectral filters, polarization elements, or a combination of the two. The filtered images are then relayed onto an image sensor. The sensor simultaneously captures an NxN array of filtered images. Note that this system allows for both polarization and multispectral measurements simultaneously. Furthermore, circular polarization can easily be measured by placing a circular analyzer in the filter array. Since the filter array is placed in the intermediate image plane, it is conveniently located to be easily swapped for a different filter array without any other modifications to the system, allowing this system to easily be reconfigured for others applications.
Note that once the relay system is constructed, any image with the necessary cone angle of light placed at the input end of the light pipe will be multiplied and filtered using this system. Therefore, the relay system and filter array are completely decoupled from the imaging objective and other optical systems.
3. Multispectral and multi-polarization imaging demonstration
A multispectral and multi-polarization system containing a 3x3 array of images was built using off-the-shelf components, as shown in Fig. 3(a). The light pipe was constructed by cutting plastic mirrors and gluing them together. In the constructed light pipe, there is significant tilt between the mirrors causing a misalignment of the reflections from perpendicular mirror surfaces. This results in an overlap of reflected images in the diagonal images in the array. An unfiltered raw image of a color wheel captured by this system can be seen in Fig. 3(b). In these diagonal images, particularly the bottom right sub-image, the tilt in the mirror surfaces creates a superposition of two images caused by reflections from two adjacent mirrors. Note that this problem can be solved if a higher precision light pipe were used in the set-up.
Fig. 3 a) Experimental system for concept demonstration. b) Raw image captured by system. c) Example of a spectral and polarization filter.
Due to the tilt in the mirror surfaces, only the non-diagonal images were used to demonstrate the imaging system in this paper. A filter array such as the one shown in Fig. 3(c) can be used to simultaneously measure spectral and polarization information. For this demonstration, the filter array of polarizing elements and spectral elements was broken up into one that contains polarization elements and one that contains spectral filters on the non-diagonal images in order to demonstrate the concept. The polarization elements (Edmund Optics Polarizer Film) used are linear polarizers and a left circular analyzer. The orientations of the linear polarizers were measured to be 0, 49, 90, and 134 degrees on the filter array. The spectral filters used are the following spectral filters from Kodak: wratten deep blue 47, wratten green 58, wratten red 25, and wratten cyan 44a. The sensor used is a monochrome sCMOS sensor with a pixel size of 6.5 microns and a sensor size of 13.3 mm.
3.1 System demonstration
In order to demonstrate the potential of this system in capturing spectral and polarization images, this system captured images of a color wheel displayed on the screen of the iPhone 5s produced by Apple, Inc. We suspect the screen on an iPhone 5s has a quarter wave linear retardation film aligned 45 degrees with respect to the plane of the linearly polarized light emitted from the screen in order to create nearly circularly polarized light on the screen [13]. We also suspect the variation of the retardation film with wavelength will create elliptically polarized light. Therefore, the flux through a polarization properties will vary with wavelength for the iPhone 5s screen. An image of a color wheel displayed on the iPhone 5s screen demonstrates its polarization properties.
For this example, the imaging objective was two 50 mm focal length lenses spaced approximately 100 mm with a stop in the middle in order to create a doubly telecentric objective. The image from this objective was formed at the input end of the light pipe and then multiplied and filtered using the relay system.
The images from the spectral filters and polarization elements are shown in Fig. 4. The individual images captured from the red, green, blue and cyan filters is shown in Fig. 4(a) colored in their respective colors. In Fig. 4(b) the red, green, and blue channels were combined in order to create a reconstructed color image of the color wheel. Raw images captured through linear polarizers rotated at 0, 90, 134 degrees and a left circular analyzer can be seen in Fig. 4(c). Ideally the third polarizer would be rotated to 135 degrees, but this discrepancy can be compensated in the mathematical calculation of the Stokes vector, so a more precise rotation was not performed. Since the polarizer isn’t exactly 135 degrees, the Stokes parameters are calculated by solving a system of linear equations. The flux from each polarization element is given by P = WS, where S is the Stokes vector and P is the flux vector obtained directly from the pixel value from the images. The rows of the matrix W consist of the top row of each polarization element’s Mueller matrix, which gives the flux through that polarization element. The Stokes vector can be calculated for every pixel using where is the pseudo-inverse of the measurement matrix for the polarization elements used [14]. The normalized Stokes vector, the degree of linear polarization (DoLP), angle of linear polarization (AoLP), and degree of polarization (DoP) were calculated for every pixel using the following definitions:
The raw polarized images captured from the linear polarizers and the normalized Stokes parameters S1, S2, and S3 are shown in Fig. 4(c). The Stokes parameters indicates that the green pixels are left-handed elliptically polarized and the red and blue pixels are right-handed elliptically polarized. Figure 4(d) displays the DoLP, AoLP and DoP. Note in the AoLP image that the red and blue portion of the image is approximately 85 degrees, while the green portion of the image has an AoLP of approximately 150 degrees. Also note that the resolution of this system is not sufficient to resolve the individual red, green, and blue pixels on the iPhone screen. Therefore, the corresponding Stokes vector for every pixel in the image is the addition of the Stokes vectors for all 3 iPhone pixels. Since the linear component of the green iPhone pixel is oriented approximately 70 degrees from the linear component of the red and blue iPhone pixels, the addition of these Stokes vector will have a very small linear component in the portions of the image in which there is a combination of green iPhone pixels with red and blue iPhone pixels, which can be seen in the calculated DoLP image. When only red and blue pixels are on or only green pixels are on, then there is no destructive interference in the linear component added to the Stokes parameters, thus the DoLP is higher. The DoP indicates that the portions of the color wheel in which there is little mixing between the green iPhone pixels and the red and blue iPhone pixels are highly polarized.Fig. 4 Spectral and polarization images of a color wheel displayed on an iPhone 5s. Note the AoLP for the green pixels are oriented differently than the red and blue pixels.
To demonstrate the issues related to the diffuser and the need for a telecentric objective, a 10 degree holographic diffuser was used to obtain the necessary cone angle of light needed for the light pipe and prevent parallax between sub-images. As can be seen in Fig. 5, the image quality is significantly degraded. Note that the spatial resolution is also significantly reduced and there are artifacts from the surface of the diffuser that are now visible in the image. Since the diffuser was not polarization maintaining, polarization images were not analyzed.
Fig. 5 Left: Spectral images of color wheel with diffuser at the input end of the light pipe. Right: Reconstructed color image using the red, green, and blue channel. Note the poor image quality.
3.2 Reconfigurability of imaging system
To demonstrate how this system can be easily reconfigured for different applications with minimal change, the imaging objective was replaced with a 0.5 NA infinity-corrected microscope objective. A 50 mm tube lens is added to create a telecentric image. To demonstrate the reconfigurability of the filter array, a polarization filter array containing only linear polarizers rotated at 0, 49, 90, and 134 degrees was used. Note that there were no other modifications to this system. For this example, a 2x2 pixel white checkerboard pattern was created using the same iPhone 5s screen imaged in the previous sections. The spectral and polarization images obtained of the checkerboard pattern are displayed in Fig. 6. The red, green, blue and cyan channels are shown in Fig. 6(a), as well as the reconstructed color image from the red, green, and blue channels in Fig. 6(b). Once again, the raw polarization images are shown, as well as the normalized Stokes parameters S1 and S2 in Fig. 6(c). Since no circular polarization elements were used in this example, the Stokes parameter S3 was ignored. The DoLP and the AoLP for every pixel is shown in Fig. 6(d). It can be seen from that the angle of linear polarization is approximately 85 degrees for the red and blue pixels and 150 degrees for the green pixels. Also note that the DoLP of the green pixel is slightly higher than the DoLP for the red and blue pixels, which also agrees with the results obtained from Fig. 4(d). A zoomed in portion of the AoLP and DoLP for a single 2x2 white square is also shown in Fig. 6(d) in order to more clearly show the opposite polarization orientations of the pixels. Note that some registration errors between pixels in the polarization images cause some polarization artifacts in the DoLP and AoLP at the edges of the red, green, and blue pixels in the image. These registration errors can be corrected by creating custom optics to minimize the distortion in each sub-image and/or applying a post-processing algorithm to remove the distortion.
Fig. 6 Spectral and polarization images of 2x2 pixel white checkerboard pattern displayed on an iPhone 5s. Note the AoLP and DoLP is different for green pixels and red and blue pixels.
One particular application where this imaging system can be used is fluorescence imaging. Shown in Fig. 7 is an image of FluoCells Prepared Slide #3 using different excitation wavelengths and the same system used in the previous microscope example. This slide was excited using a 488 nm laser to excite the green-fluorescent Alexa Fluor 488 wheat germ agglutinin, and a 533 nm laser to excite the red-orange-fluorescent Alexa Fluor 568 phallodin in this cryostat section of mouse kidney. Figure 7(a) shows the image obtained when only a 488 nm laser is used for excitation and Fig. 7(b) shows the images obtained when only a 533 nm excitation is used. Figure 7(c) shows the images obtained in each spectral channel when both 488 nm and 533 nm wavelengths are used to excite the sample. Note that fluorescent signal from Alexa Fluor 568 is detected primarily in the red channel, and the fluorescent signal from Alexa 488 is primarily detected in the green channel. A reconstructed color image of the slide when both lasers are used is shown in Fig. 7(d). Therefore, using the proper bandpass filters, multiple fluorescent signals can be captured simultaneously using this imaging system.
4. Conclusion
A snapshot multispectral and multi-polarization imaging system with telecentric optics was proposed and demonstrated using a 3x3 array of filtered images. A telecentric imaging objective is needed to maintain image quality and distribute the flux equally to each sub-image. One advantage of this system is the ability to easily reconfigure this system for multiple applications. Since the filter array is conveniently located, the filter array can easily be swapped for different types of measurements. Also, imaging objectives can easily be swapped in this imaging system without any modification to the relay system. This allows this system to easily be reconfigured for many different applications. While this system was demonstrated with a 3x3 array, it can easily be scaled up to a 5x5, 7x7, or larger array by increasing the output cone angle of the telecentric imaging objective or modifying the dimensions of the light pipe.
One caveat of this imaging system is the measured flux in each sub-image is at least 1/N2 less than the original image for an NxN array of images. Therefore, this imaging system may not be suitable for low light environments.
In the future, the custom parts can be developed to improve the performance of this system. First, a higher precision light pipe needs to be used in order to make the diagonal images in the array usable. A custom relay system can be optimized in order to reduce the distortion between sub-images, allowing for better image registration. Also, the polarization effects of the light pipe will be further characterized using polarization ray tracing. With custom elements, the images captured by this imaging system will be further enhanced.
References and links
1. J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45(22), 5453–5469 (2006). [CrossRef] [PubMed]
2. A. A. Gowen, C. P. O’Donnell, P. J. Cullen, G. Downey, and J. M. Frias, “Hyperspectral imaging – an emerging process analytical tool for food quality and safety control,” Trends Food Sci. Technol. 18(12), 590–598 (2007). [CrossRef]
3. R. M. Levenson and J. R. Mansfield, “Multispectral imaging in biology and medicine: Slices of life,” Cytometry A 69(8), 748–758 (2006). [CrossRef] [PubMed]
4. S. L. Jacques, J. C. Ramella-Roman, and K. Lee, “Imaging skin pathology with polarized light,” J. Biomed. Opt. 7(3), 329–340 (2002). [CrossRef] [PubMed]
5. S. A. Mathews, “Design and fabrication of a low-cost, multispectral imaging system,” Appl. Opt. 47(28), F71–F76 (2008). [CrossRef] [PubMed]
6. M. W. Kudenov, J. Craven-Jones, R. Aumiller, C. Vandervlugt, and E. L. Dereniak, “Faceted grating prism for a computed tomographic imaging spectrometer,” Opt. Eng. 51(4), 044002 (2012). [CrossRef]
7. A. R. Harvey, D. W. Fletcher-Holmes, A. Gorman, K. Altenbach, J. Arlt, and N. D. Read, “Spectral imaging in a snapshot,” Proc. SPIE 5694, 110–119 (2005). [CrossRef]
8. L. Gao, R. T. Kester, and T. S. Tkaczyk, “Compact image slicing spectrometer (ISS) for hyperspectral fluorescence microscopy,” Opt. Express 17(15), 12293–12308 (2009). [CrossRef] [PubMed]
9. A. Manakov, J. F. Restrepo, O. Klehm, R. Hegedus, E. Eisemann, H. Seidel, and I. Ihrke, “A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging,” ACM Trans. Graph. 32(4), 47 (2013). [CrossRef]
10. T. H. Tsai and D. J. Brady, “Coded aperture snapshot spectral polarization imaging,” Appl. Opt. 52(10), 2153–2161 (2013). [CrossRef] [PubMed]
11. M. W. Kudenov, S. Mallik, M. J. Escuti, N. Hagen, K. Oka, and E. L. Dereniak, “Snapshot imaging Mueller matrix instrument,” Proc. SPIE 8897, 88970S (2013). [CrossRef]
12. J. Kim and M. J. Escuti, “Snapshot imaging spectropolarimeter utilizing polarization gratings,” Proc. SPIE 7086, 708603 (2008). [CrossRef]
13. J. Z. Zhong, W. Chen, C. Chen, V. H. E. Yin, and S. R. Gettemy, “Display that emits circularly-polarized light,” U.S. Patent 7683983B2 (2010).
14. R. A. Chipman, “Polarimetry,” in Handbook of Optics (McGraw-Hill, 1995), pp. 22.1–22.37.
