Open Access
Add to BibSonomyAbstract
Microelectromechanical (MEMS) mirrors have extended vision capabilities onto small, low-power platforms. However, the field-of-view (FOV) of these MEMS mirrors is usually less than 90° and any increase in the MEMS mirror scanning angle has design and fabrication trade-offs in terms of power, size, speed and stability. Therefore, we need techniques to increase the scanning range while still maintaining a small form factor. In this paper we exploit our recent breakthrough that has enabled the immersion of MEMS mirrors in liquid. While allowing the MEMS to move, the liquid additionally provides a “Snell’s window” effect and enables an enlarged FOV (≈ 150°). We present an optimized MEMS mirror design and use it to demonstrate applications in extreme wide-angle structured light.
© 2016 Optical Society of America
1. Introduction
The advent of microelectromechanical (MEMS) spatial light modulators such as DMD (digital micro-mirrors) have allowed fast and accurate applications such as coded aperture [1], analysis of fast moving scenes [2] and compressive sensing [3]. These devices provide an excellent platform to extend active computer vision techniques to the low-power and miniature domains. For example, Microvision [4] offers a single raster scanning MEMS micro-mirror, which is implemented on a 5W mobile system.
In this paper we focus on the field-of-view (FOV) of these MEMS-enabled low-power vision platforms. Current designs and fabrication techniques offer MEMS optical scanning ranges of up to 90° [6]. While this is sufficient for many tasks, vision techniques that require panoramic imaging or surround-view reconstruction would require a greater viewing angle. In macro-scale systems that require a wide FOV, such as the LIDAR structured light system from [7], mechanical rotation increases the sensing range of the device. For mobile and miniature platforms, such continuous mechanical movement would place an unacceptably high power and weight cost.
To enable futuristic applications such as micro/nano LIDAR or mobile 360° coded aperture photography, wide-angle optics is required for MEMS modulators. Recently, we demonstrated a working MEMS device immersed inside a liquid whose refractive index is greater than that of air [5]. This allows the mirror to scan freely and induces the well-known ”Snell’s window” optical effect [8]. In air, such a mirror rotates over a 2θair FOV. Due to Snell’s law, a submerged mirror would have a FOV of 2θliq = 2arcsin(nsinθair). When the critical angle is reached and , the modulator’s FOV can theoretically be 180°. Figure 1 shows our prototype, where our MEMS mirror modulates an external laser.
Fig. 1 In (I) we show a ray diagram of our setup, where a MEMS mirror is immersed in liquid. We used this setup [5] to induce the well-known Snell’s window effect that allows for modulating light over a large FOV, as shown in (II). In this paper, we demonstrate, for the first time, wide-angle structured light applications using an optimized MEMS mirror that is designed for use in liquid.
In this paper, we harness and extend this MEMS breakthrough to build a wide-angle and miniature light modulator prototype for vision and imaging applications. Our contributions are:
- We design a new MEMS mirror construction that is fabricated to achieve a larger mechanical angle. This design allows nearly 150° FOV with a modulating form factor of around 1inch3, compared with 120° FOV in [5].
- We demonstrate wide-angle structured light reconstruction using a laser-based line striping vision sensor, using our proposed optical design. For the first time, we can enable reconstruction results at nearly 130° FOV and with a modulator size of around 1inch3.
Note: All the prototypes discussed in this paper use 1D scanning mirrors and require additional optics that are not miniaturized. However, this is an artificial constraint since there is no practical or theoretical limit to using 2D MEMS mirrors in our designs.
2. Related work
Our work shares connections with vision, imaging, robotics, MEMS and optical 3D printing.
Vision and computational photography
The Snell’s window effect has been used previously [9] for a passive vision sensor using a refractive slab. In contrast, we use liquid media that allows mechanical motion inside the optics and enables both passive and active sensing. It is worth noting that MEMS and other spatial light modulators have been widely used in vision and imaging as programmable templates [1, 10]. While we use a single MEMS sensor in this work, much of our optical designs will carry over to such devices. Curved optical designs can enhance the FOV of our devices, and our ideas complement work in computational photography to increase imaging bandwidth using curved sensors [11–13] and artificial insect eyes [14, 15].
MEMS mirrors
Since the first MEMS scanning mirror by K. E. Peterson [16], MEMS mirrors have enabled applications such as optical coherence tomography [17], projectors and displays [18] and printers [19]. Wide-angle MEMS mirrors in air include different driving mechanisms such as electrostatic [20, 21], piezoelectric [22], electromagnetic [23], and electrothermal [24]. Some of these technologies have problems when operating in liquid. For example, due to viscosity damping, most resonance-based electrostatic or piezoelectric mirrors may have significant decrease in performance. We focus on electrothermal devices since electromagnetic mirrors require magnets or coils which increase fabrication complexity. In addition, electrothermal devices have a large spacing underneath the mirror, which allows liquid to fully surround the mirror and prevents air bubbles from forming.
Although Huang et al. [23] investigated the impact of acoustic MEMS devices in water, we were the first [5] to show that MEMS mirrors immersed in liquid could scan over a large range at moderate speed and could also have interesting control properties such as voltage, temperature gradient, etc. In particular, we showed an increased optical viewing angle is possible. In this paper we propose a design with a wider FOV than [5] and we are the first to use MEMS mirrors immersed in liquid for structured light reconstruction.
Other miniature wide-FOV optics
Wide FOV optics [25] and micro-optics are active research fields [26–28]. Our liquid MEMS design offers a new axis of optical modulation when compared to other available options. For example, commercially available fish lenses (such as [29]) and those custom made for research purposes (e.g. [30]) consist of multiple optical elements at cm scales. However, using these for MEMS modulation, as done by [30], has resulted in only 120° FOV being demonstrated. Miniature polygonal mirror wheels allow fast laser modulation for a variety of applications such as range scanning [31]. However, when compared with MEMS devices, polygonal mirrors face a disadvantage with respect to raster scanning; 2D MEMS mirrors can switch very quickly to reflect light in different directions, which is not possible with a polygonal mirror wheel. Curved mirrors allow panoramic imaging for computer vision applications [32] and have no dispersion related problems. Some designs, such as hyperbolic and ellipsoidal mirrors, have single-viewpoints, which allow many computational advantages. Unfortunately, to the best of our knowledge, the state-of-the-art for miniature curved mirrors does not appear to have a greater FOV than 45° [33].
3. MEMS mirror immersed in liquid
Electrothermal lateral shift free (LSF) mirrors with three bimorph segments can cancel the lateral shift and achieve large actuation range [24]. We created a device using this principle, shown in Fig. 2(I–II). The fabrication process for this device is the same as reported in [24]. The large initial elevation and separation of the bimorph beams make the mirror less likely to suffer from the damping or stiction problems that occur in a liquid medium. In addition, as the resonance cannot be excited when the MEMS mirror scans in liquid with large viscosity, the actuation mechanism is required to be able to generate large enough scan range at non-resonance mode, which is exactly the advantage of electrothermal actuation.
Fig. 2 Our MEMS device design (I) is imaged in (II). We show a photograph of the packaged MEMS device in (III). Using a laser setup similar to Fig. 1, we measured the optical angle scanned (using an optical table and graph paper) for multiple MEMS mirror positions (IV), given by the voltage applied to the MEMS device. We performed this experiment in air and then in mineral oil (n = 1.5). We were able to obtain close to 150° FOV with the use of mineral oil, due to the Snell’s window effect. In (V) we show a long exposure image of the device as it modulates the laser in three distinct directions, illustrating the wide FOV.
Only [5] has demonstrated immersing a MEMS mirror in liquid. Compared with [5] the embedded heater Pt layer in our design is increased from 0.1μm to 0.2μm while the active Al and SiO2 layers of the bimorphs are still kept at 1μm, as in Fig. 2(I). The thickness increase strengthens the bimorphs which increases the maximum achievable mechanical angle. As the amplification of angle increases drastically when the mechanical angle is approaching the critical angle (21.5°), the actual mechanical angle reached in this new device is about 20° compared with 18° of [5], resulting in the 150° optical scanning range compared with 120°. Moreover, with the increase of Pt thickness, the resistance of embedded heater changes from about 80Ω to 40Ω, thus the actuation voltage needed to deliver the same amount of electrical power to the bimorphs is reduced, we can reach about 150° with less than 7V, compared 120° with 10V reported in [5].
The footprint of the LSF mirror is 3.2mm×3.2mm with the mirror plate size of 1.7mm×1.8mm. Two sets of LSF actuators are symmetrically arranged on both sides of the mirror plate along one axis, as shown in Fig. 2(II). With differential control of each actuator, the mirror plate can scan as a seesaw, and thus a symmetric scan range can be achieved.
For packaging, the MEMS mirror is first glued to a DIP package and then wire bonding is used to provide electrical connections. Meanwhile, a plastic holder is 3D printed with small through holes that the pins of the DIP package can fit in. After the DIP package is aligned and the pins are pressed through the holes, the hole regions are hermetically sealed to prevent the liquid from leakage. After that, a glass slide is slid through from one side and functions as the front optical window. After the glass slide is well sealed, the liquid is slowly filled into the chamber formed by the plastic holder and the glass slide. We utilize mineral oil for the surrounding liquid, as in Fig. 1, since it does not conduct electricity and since it has a high refractive index. Finally, the entire assembly is placed into a vacuum chamber to remove most of the air dissolved in the liquid. The final packaged device is shown in shown in Fig. 2(III).
Once immersed in the oil, the same mechanical rotation is able to achieve a ±75° optical scanning range at only 6.7V. We measured the fields of view for different MEMS locations using the setup in Fig. 1 for both air and liquid immersion, and these are reported in Fig. 2(IV). Our device demonstrates a field-of-view of 150° depicted in Fig. 2(V) using a long exposure.
4. Wide-angle structured light
We use MEMS mirror design to modulate a laser for structured light line striping [34]. Since our prototype uses a 1D scanning mirror that modulates a single laser, we use a custom cylindrical lens of 6mm focal length to convert the laser spot into a light plane. Our ray diagram and optical setup is given in Fig. 3 (I–III), where the MEMS sensor is packaged in mineral oil. We use a 1mm thick glass slide (whose refractive index of 1.46 is similar to that of mineral oil) and a 3D printed part made with an Objet Connex 20 micron plastic printer to form the housing of the MEMS mirror and the liquid. This design uses gravity to ensure that there are no air pockets on the liquid-glass boundary and that the liquid-air boundary is not in the optical path. In Fig. 3 (IV) we show an image captured from our setup, revealing a bright laser line stripe.
Fig. 3 (I) shows a diagram of the setup in (II), which allows us to scan a laser line across a 150° FOV, due to the refractive effect of the mineral oil shown in (III). In (IV) a scene consisting of cardboard boxes is shown with the projected laser line stripe.
As our device sweeps a laser light plane over the scene, we use spacetime stereo [35, 36] to exploit the illumination variation to recover horizontal disparity. We use a Point Grey Bumblebee2 stereo camera that provides calibrated, synchronized and rectified video streams to obtain depth from disparity. As a review, spacetime-stereo matches the intensity profile of a pixel in one camera to the closest match along the epipolar line in the other camera. Therefore, if we are considering pixel (x, y) in the left camera image Il, then the search optimization becomes,
where d is the stereo disparity, t is the variable corresponding to the video time sampling and the epipolar line is horizontal since the images are rectified.We place a scene consisting of four boxes of sizes, ranging from a few inches to approximately a foot in width, in front a white lambertian plane such that each box is at a random distance from the plane. The scene is then moved to different positions in the field of view of the sensor; first at 46° from the normal viewing direction, as in Fig. 4(I), and then at 64° from the normal, as in Fig. 4(III). The latter represents a 128° sensor FOV, which is lower than the achievable 150° FOV due to limited size of our optical dark room. Our optical setup is in exactly the same location in the two experiments. What changes is the scene location and the camera positions that measure image data. We demonstrate reasonably good depth estimates both qualitatively from the depth maps in the figure and quantitatively, by measuring the inter-box distance using a tape measure and comparing with results from the stereo reconstruction. The latter revealed a ±2 inches error over the entire scene depth of 30 inches. The depth maps provide a correct reconstruction of the scene, as demonstrated by the rendered point cloud, and are consistent across the FOV. Since the boxes were fronto-parallel we also calculated the RMS distance by fitting a plane to each box and this average error was 0.56 inches, which is confirmed by the mostly error free depth map.
Fig. 4 In (I) the cardboard scene shown in Fig. 3 is kept at a “close position” within a 92° FOV. In (II) the same scene is moved to a “far position”, and is still illuminated by our sensor at around 128° FOV. Note that our sensor has a 150° FOV and our darkroom was not big enough (at that scene depth) to demonstrate the full FOV of the device. In (II) and (IV) we use a stereo camera viewing the line striped scene to recover depths. The gray color bar shows the relative depths with respect to the rear plane. We obtained an average error of ±2 inches over the entire scene depth of 30 inches. Since each of the five boxes were fronto-parallel, we also calculated the RMS distance by fitting a plane to each box and the average such error was 0.56 inches.
Finally, we applied the same method to reconstruct a toy doll, as shown in Fig. 5. The doll was placed approximately at 50° from the normal direction. In (I) we show a color image of the doll, demonstrating that it consists of non-lambertian materials, such as fur and plastic. In (II) we show a depth map from our method, without any post-processing, which is almost noise-free. A rendering from a different angle in (III) was created by estimating surface normals and rendering under different lighting. Note that the thumb of the doll’s right hand and the stub of its toes are clearly visible in the reconstructed result.
Fig. 5 We use the setup in Fig. 4 to reconstruct a toy doll (I). The object was placed around 50° from the viewing direction. We show the depth map (II) and a 3D reconstruction (III) that is rendered from a different viewpoint. Note that the color image was not taken from either stereo pair but from an SLR camera placed close to the stereo pair.
5. Limitations for fast scanning
The frequency responses of the MEMS mirror working in both the air and the mineral oil are shown in Fig. 6(I) and (II) respectively. Due to the introduction of surrounding liquid, when the MEMS mirror is immersed, the frequency response differs from the case of in the air. For the electrothermal bimorph actuators, the low frequency response is dominated by the thermal domain. With the increase of frequency, the thermal response will have magnitude drop due to the low thermal cut-off frequency [37]. As shown in the figure, the 3 dB cut-off frequencies for the MEMS mirror are 4 Hz and 2 Hz for the cases in the air and the mineral oil, respectively. The decrease of thermal cut-off frequency in the mineral oil is believed to be cause by the induced large thermal capacitance around the thermal bimorph actuators [5]. The tip-tilt mode resonant frequency of the MEMS mirror when working in the air is about 422 Hz, however, when the MEMS mirror is immersed in the mineral oil, the scanning of the mirror plate suffers from the large viscosity and the resonance cannot be excited. Thus, due to the low thermal cut-off frequency and the large viscosity, the MEMS mirror cannot scan with fast speed when immersed in the mineral oil. As discussed previously, the immersing effect enlarges the scan range by twice, with this advantage, the maximum scan range of the MEMS mirror in the mineral oil is still larger than the case of in the air for scan frequency of up to 70 Hz. This implies that our design can be used for applications where the scan speed may not be crucial, such as slow moving scenes.
6. Summary and future work
In summary, we have presented a new MEMS mirror design that allows for a large scanning angle and have demonstrated its ability to move in liquid and modulate light over a large FOV. This allowed for the construction of a line striping projector with almost a 150° FOV in a small, compact form. The main challenge we have overcome is to enable, for the first time, vision applications for a MEMS mirror modulating light while remaining fully submerged in liquid. This opens up a new range of questions of optical design, MEMS engineering and vision sensing. An important issue that we face is the decrease in bandwidth of the MEMS scanning due to being inside the liquid. In this paper we have avoided this problem by considering static scenes. For dynamic scenes, increased MEMS motion will result in the issues of heat transfer inside the liquid, turbulence and the relationship of these physical effects to motion blur. We hope to explore these types of research questions in the future.
Acknowledgments
This research has been partially supported by funding from the National Science Foundation (NSF) with Grant #1514154.
References and links
1. S. K. Nayar, V. Branzoi, and T. E. Boult, “Programmable imaging: Towards a flexible camera,” International Journal of Computer Vision 70, 7–22 (2006). [CrossRef]
2. S. J. Koppal, S. Yamazaki, and S. G. Narasimhan, “Exploiting dlp illumination dithering for reconstruction and photography of high-speed scenes,” International journal of computer vision 96, 125–144 (2012). [CrossRef]
3. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. E. Kelly, R. G. Baraniuk, et al., “Single-pixel imaging via compressive sampling,” IEEE Signal Processing Magazine 25, 83 (2008). [CrossRef]
4. Microvision, “Picop technology,”www.microvision.com (2015).
5. X. Zhang, R. Zhang, S. Koppal, L. Butler, X. Cheng, and H. Xie, “Mems mirrors submerged in liquid for wide-angle scanning,” in “Solid-State Sensors, Actuators and Microsystems (TRANSDUCERS), 2015 Transducers-2015 18th International Conference on,” (IEEE, 2015), pp. 847–850.
6. S. T. Holmstrom, U. Baran, and H. Urey, “Mems laser scanners: a review,” Microelectromechanical Systems, Journal of 23, 259–275 (2014). [CrossRef]
7. Velodyne, “Velodyne lidar,”, velodynelidar.com (2015).
8. R. W. Wood, Physical optics (The Macmillan Company, 1905).
9. S. J. Koppal, I. Gkioulekas, T. Young, H. Park, K. B. Crozier, G. L. Barrows, and T. Zickler, “Toward wide-angle microvision sensors,” Pattern Analysis and Machine Intelligence, IEEE Transactions on 35, 2982–2996 (2013). [CrossRef]
10. A. Zomet and S. K. Nayar, “Lensless imaging with a controllable aperture,” in “Computer Vision and Pattern Recognition, 2006” IEEE Computer Society Conference on,”, vol. 1 (IEEE, 2006), vol. 1, pp. 339–346.
11. O. S. Cossairt, D. Miau, and S. K. Nayar, “Scaling law for computational imaging using spherical optics,” JOSA A 28, 2540–2553 (2011). [CrossRef] [PubMed]
12. D. Brady, M. Gehm, R. Stack, D. Marks, D. Kittle, D. Golish, E. Vera, and S. Feller, “Multiscale gigapixel photography,” Nature 486, 386–389 (2012). [CrossRef] [PubMed]
13. H. C. Ko, G. Shin, S. Wang, M. P. Stoykovich, J. W. Lee, D.-H. Kim, J. S. Ha, Y. Huang, K.-C. Hwang, and J. A. Rogers, “Curvilinear electronics formed using silicon membrane circuits and elastomeric transfer elements,” Small 5, 2703–2709 (2009). [CrossRef] [PubMed]
14. K.-H. Jeong, J. Kim, and L. P. Lee, “Biologically inspired artificial compound eyes,” Science 312, 557–561 (2006). [CrossRef] [PubMed]
15. S. Hiura, A. Mohan, and R. Raskar, “Krill-eye: Superposition compound eye for wide-angle imaging via grin lenses,” in “Computer Vision Workshops (ICCV Workshops), 2009 IEEE 12th International Conference on,” (IEEE, 2009), pp. 2204–2211.
16. K. E. Petersen, “Silicon torsional scanning mirror,” IBM Journal of Research and Development 24, 631–637 (1980). [CrossRef]
17. Y. Pan, H. Xie, and G. K. Fedder, “Endoscopic optical coherence tomography based on a microelectromechanical mirror,” Optics letters 26, 1966–1968 (2001). [CrossRef]
18. H. Urey, D. W. Wine, and J. R. Lewis, “Scanner design and resolution trade-offs for miniature scanning displays,” in “Electronic Imaging’99,” (International Society for Optics and Photonics, 1999), pp. 60–68.
19. W. O. Davis, D. Brown, M. Helsel, R. Sprague, G. Gibson, A. Yalcinkaya, and H. Urey, “High-performance silicon scanning mirror for laser printing,” in “MOEMS-MEMS 2007 Micro and Nanofabrication,” (International Society for Optics and Photonics, 2007), pp. 64660D. [CrossRef]
20. A. C.-L. Hung, H. Y.-H. Lai, T.-W. Lin, S.-G. Fu, and M. S.-C. Lu, “An electrostatically driven 2d micro-scanning mirror with capacitive sensing for projection display,” Sensors and Actuators A: Physical 222, 122–129 (2015). [CrossRef]
21. Y. Liu, J. Xu, S. Zhong, and Y. Wu, “Large size mems scanning mirror with vertical comb drive for tunable optical filter,” Optics and Lasers in Engineering 51, 54–60 (2013). [CrossRef]
22. C.-D. Chen, Y.-J. Wang, and P. Chang, “A novel two-axis mems scanning mirror with a pzt actuator for laser scanning projection,” Optics express 20, 27003–27017 (2012). [CrossRef] [PubMed]
23. C.-H. Huang, J. Yao, L. V. Wang, and J. Zou, “A water-immersible 2-axis scanning mirror microsystem for ultrasound andha photoacoustic microscopic imaging applications,” Microsystem technologies 19, 577–582 (2013). [CrossRef]
24. L. Wu and H. Xie, “A large vertical displacement electrothermal bimorph microactuator with very small lateral shift,” Sensors and Actuators A: Physical 145, 371–379 (2008). [CrossRef]
25. K. Miyamoto, “Fish eye lens,” JOSA 54, 1060–1061 (1964). [CrossRef]
26. H. P. Herzig, Micro-optics: elements, systems and applications (CRC Press, 1997).
27. R. Völkel, M. Eisner, and K. Weible, “Miniaturized imaging systems,” Microelectronic Engineering 67, 461–472 (2003). [CrossRef]
28. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, “Thin observation module by bound optics (tombo): concept and experimental verification,” Applied optics 40, 1806–1813 (2001). [CrossRef]
29. S. D. I. optics, “Dsl221 specifications sheet,” www.sunnex.com (2015).
30. V. Milanović, K. Castelino, and D. T. McCormick, “Highly adaptable mems-based display with wide projection angle,” in “Micro Electro Mechanical Systems, 2007. MEMS. IEEE 20th International Conference on,” (IEEE, 2007), pp. 143–146.
31. R. J. Sherman, “Polygonal scanners,” Laser Beam Scanning, pp. 63–75 (CRC Press, 1985).
32. S. Baker and S. K. Nayar, “A theory of single-viewpoint catadioptric image formation,” International Journal of Computer Vision 35, 175–196 (1999). [CrossRef]
33. C. Gimkiewicz, C. Urban, E. Innerhofer, P. Ferrat, S. Neukom, G. Vanstraelen, and P. Seitz, “Ultra-miniature catadioptrical system for an omnidirectional camera,” in “Photonics Europe,” (International Society for Optics and Photonics, 2008), pp. 69920J. [CrossRef]
34. J.-Y. Bouguet and P. Perona, “3d photography using shadows in dual-space geometry,” International Journal of Computer Vision 35, 129–149 (1999). [CrossRef]
35. J. Davis, R. Ramamoorthi, and S. Rusinkiewicz, “Spacetime stereo: A unifying framework for depth from triangulation,” in “Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on,”, vol. 2 (IEEE, 2003), vol. 2, pp. II-359.
36. L. Zhang, B. Curless, and S. M. Seitz, “Spacetime stereo: Shape recovery for dynamic scenes,” in “Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on,”, vol. 2 (IEEE, 2003), vol. 2, pp. II-367.
37. S. Pal and H. Xie, “A parametric dynamic compact thermal model of an electrothermally actuated micromirror,” Journal of Micromechanics and Microengineering 19, 065007 (2009). [CrossRef]
