Open Access
Add to BibSonomy
Abstract
We propose a compact form of the triple-channel panoramic annular lens (PAL) with a stereo field and no central blind area based on polarization technology, which solves the problem that the traditional stereo panoramic system always has a large and complex mirror in front to reflect light. Based on the traditional dual-channel structure, we apply polarization technology to the first reflective surface to create a third channel for the stereovision. The field of view (FoV) of the front channel is 360° × (0° − 40°), the FoV of the side channel is 360° × (40° − 105°) and the stereo FoV is 360° × (20° − 50°). The Airy radii of the front channel, the side channel, and the stereo channel are 3.374 µm, 3.372 µm, and 3.360 µm, respectively. The modulation transfer function at 147 lp/mm is greater than 0.13 in the front and stereo channels and greater than 0.42 in the side channel. The F − θ distortion of all FoVs is less than 10%. This system shows a promising way to achieve stereovision without adding complex structures on the original basis.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
In recent years, with the development of network transmission technology and artificial intelligence, the traditional imaging system has gradually failed to meet the requirements of machine vision, intelligent identification, automatic driving, and other fields. Panoramic annular lens (PAL), a special ultra-wide-angle optical system, can capture the 360$^\circ$ ambient light around the system to obtain extensive information. Compared to the conventional panoramic system, it has a wider field, higher resolution, and less distortion on the marginal field of view (FoV). Due to the above advantages, it has great potential in the future development [1–6], especially in the multidimensional information perception field, where PAL is expected to capture a great deal of depth image information of objects in the panoramic FoV.
Existing panoramic imaging systems mainly include single sensor scanning type, single sensor gazing type, multi-sensor stitching type, et al. [7,8]. On this basis, to obtain 3D information in panoramic view, which is important to machine visualization and distance measurement, researchers have designed a variety of single-sensor gazing panoramic stereo imaging systems with refractive or reflective structures and PALs. Researchers have used two fisheye lenses and other cameras to achieve stereo imaging [9]. A panoramic system with a reflective mirror in the front and a fisheye lens in the back is proposed. The former of the mirror converts the rays from the back to enter the fisheye lens and the latter of the fisheye lens captures the rays from the front FoV [10]. But due to the large distortion on the marginal FoV of the fisheye lens, it has a limited resolution. Because of the unique imaging characteristics of PAL, Wang et al. replaced the fisheye lens with a PAL to achieve less distortion and a very large panoramic FoV [11]. Huang et al. added another PAL block in front of the traditional panoramic system to achieve a stereo field of 45$^\circ$ and it also eliminated the central blind area by converting the stereo rays from the former PAL block to the original blind area of the latter PAL [12].
To sum up, the existing panoramic stereo imaging system usually has a large and irregular mirror in the front, whose irregular surface is too complex and expensive to manufacture, and it demands very strict assembly accuracy. Besides, for the design combining mirror and PAL or two PALs, it still has a central blind area that can not capture the front field or they fill the blind area with the stereo field instead of the front field. However, some researchers have proposed panoramic systems without central blind area by using dichroic films [13] or polarizing plate beam splitter [14]. To combine the advantages of the above systems and to eliminate the central blind area to capture the front field and has a stereo field at the same time, we apply the polarization technology on the first reflective surface instead of the second reflective surface based on traditional no-blind-area panoramic systems.
In this work, a real-time triple-channel panoramic stereo imaging system with no central blind area and no large front unit is proposed. The designed system has front, stereo, and side channels with FoVs of $360^{\circ }{\times }(0^{\circ }-40^{\circ })$, $360^{\circ }{\times }(20^{\circ }-50^{\circ })$, $360^{\circ }{\times }(40^{\circ }-105^{\circ })$, respectively and the stereo FoV is $360^{\circ }{\times }(20^{\circ }-50^{\circ })$. Compared to other panoramic stereo systems, it has four advantages. First, all three channels can be focused on the same image surface without extra mechanical structure to adjust the back focus. Second, it does not require a large mirror or another PAL block to form the stereovision. Thus, it is much more compact and smaller in the front. Third, it has a panoramic stereo field and no central blind area, improving the utilization rate of the sensor. Forth, to the best of our knowledge, it is the first panoramic stereo system that utilizes the first reflective surface as a shared surface to achieve multi-channel.
2. Design principle of the triple-channel panoramic stereo imaging system
2.1 Imaging principle
The PAL system is inspired by a scallop whose visual system has over 200 eyes consisting of a refractive lens and a concave mirror to form the special refraction and reflection light path. It is first proposed by Greguss in 1986 [15]. As is shown in Fig. 1, the PAL block first converts the rays into the lens and then reflects them twice on the first and second reflective surface to make the angle of the light much smaller. The relay lens group manages to eliminate aberrations and form a clear image. The PAL system follows the flat cylinder projection principle [16], which transforms the cylinder FoV into the annular image on the sensor. Although it has a super wide FoV, there are many aberrations in the system, such as distortion and field curvature and it has a low utilization rate of the sensor due to the central blind area. In addition, it has a large PAL block, which mainly turns the lateral distance into the longitudinal distance by the reflective surfaces. As for the stereo panoramic systems, they still have the above problems especially the absence of the front FoV and the complex structure.
The proposed design is composed of a PAL block and a front lens as shown in Fig. 1. A front lens is placed in the front of the PAL block along the optical axis and there is another lens group after the PAL block. The front lens is used to receive the rays from the front channel, which eliminates the central blind area. The PAL block is designed to convert the rays from the side and stereo channel, whose aberrations, such as distortion, coma, and chromatic aberration, can be mainly corrected by the relay lens group. There are three optical paths emitted from the object point, which can be imaged to different point positions of the sensor respectively. The red line represents the rays that go through the front lens, the PAL block, and the relay lens group. The blue line means that the rays directly go through the PAL block and the relay lens group. And the black line indicates the rays go through the same lens as the blue line but reflect on the back and the front surface of the PAL block and then be imaged to the sensor. As is shown, the FoV of the system has the front, the stereo, and the side channel. The vertical angle of the above is from $\alpha _1$ to $\alpha _2$, from $\beta _1$ to $\beta _2$ and from $\gamma _1$ to $\gamma _2$, respectively. It should be noted that $\alpha _2$ equals $\gamma _1$, which means the overlapping field from $\beta _1$ to $\beta _2$ all can be used for the stereovision.
For a more in-depth explanation of the triple-channel design, the propagation of the S and P components are presented in Fig. 2. Since the wire grid polarizing film [17] reflects the S component and transmits the P component, the orange line, which represents the rays from the stereo channel, will be split into two states of polarization and only the P component can pass through the system. As for the green line representing the rays from the side channel, the S component of it will reflect and refract to pass the PAL block as normal panoramic rays. An annular film is used to make sure the rays from the front channel pass and the side and stereo channel noninterference, which will be discussed in detail later. The design of the shared back surface of the PAL block saves a considerable amount of space and indicates a new way of a multi-channel panoramic system.
2.2 Calculation principle of the optical paths of the triple-channel
Because of the usage of the wire grid polarizing film, the optical paths of the stereo channel and the side channel must be calculated accurately to make sure they do not interfere with each other. For the side channel, the S component of the rays will reflect on the wire grid polarizing film and reflect again on the first surface of the PAL block coated with reflective film. Then the rays will go through the back surface of the PAL block. It has two issues to consider. First, on the front surface, it should have a part of the area without reflective film for the front channel to pass. Second, on the back surface, the S component of the side channel can not transmit on the second time through this surface. Therefore, it has to leave a certain area without wire grid polarizing film for the front and side channel to pass. To sum up, it is very important to calculate the paths of the three channels to arrange the film layer area reasonably.
As is shown in Fig. 3, the origin of the coordinates is point O, the vertex of the first surface. We set the radius of the front surface as $r_1$, and the refractive surface of the PAL block as $r_3$, the reflective surfaces of the PAL block as $r_4$ and $r_5$, respectively. Then we set the thickness of the front lens as $t_1$, the distance between the back surface of the front lens, and $r_3$ as $t_2$. And there is a small distance between $r_3$ and $r_4$, which is set as $t_3$. As for the PAL block, the distance between $r_4$ and $r_5$ is set as $t_4$. It is necessary to calculate the paths of the maximum FoV of the front channel colored red and the minimum FoV of the side channel colored black in Fig. 3. If these two optical paths have no interference, the two channels will work as expected.
For the red line on the right in Fig. 3, since it is the maximum FoV of the front channel, $A$ is a point on the lateral diameter of the front lens, so its $y$-axis coordinate can be written as Eq. (1).
And the $z$-axis of $A$ is determined by the equation of the surface $r_1$, which is written as Eq. (2)
As is shown, the slope of the line $AB$ can be represented by the half of the maximum FoV as Eq. (3)
Then expressions of line $AB$ and surface $r_2$ can be written as Eq. (4).
The coordinate of $B$ can be determined using the above expressions. We set the coordinate of $B$ as $y_B$ and $z_B$, therefore the normal line of surface $r_2$ on point $B$ can be represented as Eq. (5).
And the incident angle $\theta _2$ can be represented as Eq. (6)
According to Snell’s law, the refraction angle $\theta _2'$ is Eq. (7).
Therefore, the slope of the line $BC$ can be calculated by the Eq. (5) – (7). And with the slope of the line $BC$ and the equation of the surface $r_4$, the coordinate of $C$ is determined. Using the same mathematical calculations as above, we have the coordinate of $S_1$ and the slope of line $CS_1$. To avoid interference between the channels, the $y$-axis of point $C'$ must be less than point $C$, and the slope of line $C'S_2$ must be bigger than line $CS_1$. According to the above two boundary conditions and $\omega _2$, the minimum of the FoV of the side channel, we can calculate backward to get the coordinate of the incident point $A'$. Based on the above calculation, the front surface of the PAL block ranging from center to point $C$ will have no reflective coating to let the front channel pass and from point $C'$ to $A'$ will have the reflective coating to reflect the side channel.
As is shown in Fig. 4, there are also some boundary conditions to avoid interference between the side and the stereo channel. All the boundary conditions are listed as Eq. (8).
As is shown in Fig. 5, there are three parts of the image surface. The green part is the image area of the front channel. The red part is the image area of the side channel. And the purple part is the image area of the stereo channel. The radii of the three parts can be represented as Eq. (9).
The point $P_1$ and $P_2$ in the stereo channel have two image points on two different channels. According to the length of the two image points, the 3-D information of the point can be calculated. The preferable distribution of the three channels is that the diameter of the stereo channel equals 1.47 times the diameter of the other two channels, due to the $f-\theta$ distortion of the large angle systems [12]. But in practical optical design, it is difficult to achieve that. Therefore, the diameter of the stereo channel manages to equal 1.96 times the diameter of the other two channels after several trials.
3. Optimization process
The design optimization process is shown in Fig. 6. In general, the front channel has an extra front lens compared to the side channel, which makes it easier to design. Therefore, we first designed the side and stereo channels. The process can be divided into three parts. First, according to the above calculation, we calculated the structure of the PAL block to make sure the rays from the side and stereo channels pass through the system. Then, we designed the relay lens group. To separate the two channels on the image plane, we managed to control the angle of the chief rays in order at the stop and the back surface of the PAL block. And we calculated the structure of the relay lens group after the stop and the front lens. Finally, we optimized the image quality of the system and analyze the tolerance of the system.
There are three main aberrations in the optimization process. Since the system has three channels and a large field, the stray light has to be considered. Huang et al. analyzed the stray light of the PAL system in detail and listed some suppression methods to decrease it [18]. The normal PAL block is usually composed of a cemented doublet, whose mid-surface may split the rays and bring an amount of stray light. However, our PAL block is a single lens without that kind of stray light. And setting the stop at the back surface of the PAL block or far from the PAL block are two main ways to reduce the stray light according to the above article. Considering the shared usage of the wire grid polarizing film on the back surface, we choose to set the stop far from the PAL block to suppress the stray light instead of setting the stop on the back surface.
For the multi-channel system with different focal lengths in each channel, the back focus may be different, especially with the utilization of the shared surface for the overlapping FoV. We manage to control the angle of the rays out of the PAL block to convert the multi-channel rays into one. As is shown in Fig. 7, we convert the rays from three channels with overlapping parts into one combined unit within 100$^\circ$. Therefore, the relay lens group can be taken as one channel system with a field of 100$^\circ$, which can be simple to ensure the back focus is a constant. Another aberration is chromatic aberration, which we used two cemented doublets to correct [19].
4. Image performance of the triple-channel system
According to the above calculation, a CMOS image sensor with 3.4$\mathrm{\mu}$m pixel size is chosen. In Table 1, we list the detailed parameters of the three channels. The designed system has a short total length and a high resolution with good image quality. The whole system with three channels can be seen in Fig. 8
Table 1. The specific parameters of the three channels
4.1 Side and stereo channel
Different from other PAL blocks, this system has a side and stereo channel, which makes it a meniscus instead of a biconvex lens, as shown in Fig. 9. It can convert the rays from the stereo channel as a fisheye lens, making sure the two image points of the object image on the same side of the image surface and it is much smaller. We use the above ray tracing method to calculate the radii and make sure the two channels do not interfere. As we can see, the front surface can be divided into two parts. One is the refractive surface that converts the side channel rays. The other is the reflective surface as the second reflective surface of the PAL block. And on the back surface, there are three kinds of rays. The reflect rays $A$ come from the side channel, which is the S component. And the refractive rays $B$ come from the stereo channel, which is the P component. As for the refractive rays $A'$, they share the area with the front channel rays where is no wire grid polarizing film. Then to optimize the aberrations of the channels and convert the rays with a large incident angle, we set the stop to 30.99 mm from the PAL block.
The structure of the side channel is shown in Fig. 10. The maximum vertical FoV of the side channel is 105$^\circ$. The focal length is 0.284 mm, the maximum diameter is 48 mm and the total length is 65.95 mm. Considering the cost and difficulty of manufacturing, all lenses are spherical. And based on the pixel of the sensor, the Nyquist cutoff frequency can be determined as 147 lp/mm. As is shown in Fig. 11, the RMS spot radii are smaller than 1.750 $\mathrm{\mu}$m and the ray aberrations are less than $\pm 10$ $\mathrm{\mu}$m. As is shown in Fig. 12, the MTF is higher than 0.428 at 147 lp/mm and the $F-\theta$ distortion is less than 10$\%$. The existence of the distortion can increase the relative illuminance [20] to improve the imaging result and the distortion can be eliminated by the algorithm. The image quality of the side channel meets the requirements of panoramic stereo imaging.
The structure of the stereo channel is shown in Fig. 13. The FoV of the stereo channel is $360^{\circ }{\times }(20^{\circ }-50^{\circ })$. The effective focal length is 4.688 mm and the total length is the same as the side channel. As is shown in Fig. 14, the RMS spot radii are smaller than 3.658 $\mathrm{\mu}$m and the ray aberrations are less than $\pm 10$ $\mathrm{\mu}$m. As is shown in Fig. 15, the MTF is higher than 0.132 at 147 lp/mm and the $F-\theta$ distortion is 6.987$\%$. It can be seen as a fisheye lens, which is capable of imaging a large FoV as stereovision but not too large a field to produce large distortion. In this way, this system solves the problem generated by mixing the reflective mirror and fisheye lens.
4.2 Front channel
Based on the completed structure of the side and stereo channel, the front channel is designed. As is shown in Fig. 16, the PAL block in the front channel acts as a normal lens. The FoV of the front channel is $360^{\circ }{\times }(0^{\circ }-40^{\circ })$. The effective focal length is 4.707 mm and the total length is 81.114 mm. As is shown in Fig. 17, the RMS spot radii are smaller than 2.864 $\mathrm{\mu}$m and the ray aberrations are less than $\pm 10$ $\mathrm{\mu}$m. As is shown in Fig. 18, the MTF is higher than 0.147 at 147 lp/mm and the $F-\theta$ distortion is 9.933$\%$. In this system, the front lens group is much smaller and has no central blind area of the field.
4.3 Tolerance analysis
The difficulty of processing and manufacturing is an important subject in optical design. A qualified optical system should have a low tolerance sensitivity and a less complex surface shape to reduce the difficulty of assembly. Table 2 indicates the tolerance values of the design system. We select the diffraction MTF at 147 lp/mm of the system as the evaluation standard. As is shown in Fig. 19, the MTF is higher than 0.1 in all FoVs with the tolerance given above. And there is no aspheric lens or other complex lenses, which makes this system easier to process and manufacture.
Table 2. Tolerance values of the system
5. Depth extracting analysis
The accuracy of the depth information extraction is mainly affected by the image quality and the focal length. For the image quality, too high distortion significantly affects the depth information because the deformed image leads to the incorrect judgment of the object shape and it causes great difficulties for system calibration. From an algorithmic point of view, the distortion can be mainly corrected by the algorithm [21,22] and the results show that the distortion can be reduced to approximately 0.5%. And from the perspective of the optical design, Zhou et al. proposed a PAL system with 0.5% $F-\theta$ distortion [23], which can avoid the usage of the algorithm. As for the focal length, the longer the focal length, the longer the baseline length will be, resulting in more accurate depth information. Wang et al. proposed an extra long focal length panoramic lens based on ogive and aspheric surface [24]. But it is difficult to design a panoramic system with low distortion and long focal length, especially for the stereo system we proposed. Therefore, after weighing all the factors, we retained the large field, spherical surface design, and compact structure instead of the low distortion and long focal length.
For stereo-visual analysis, the depth information of objects is determined by the baseline length and the pixel size. From the theoretical level, we roughly calculated the baseline length. With the triangulation method in the distance measuring, the baseline length is 6.288 mm between the side and stereo channel, and 10.992 mm between the front and stereo channel. The least observable displacement on the image plane is determined by the pixel size of the system, which also affects the least depth and angle error. According to the specific parameters of the system, the least angle error is 0.143$^\circ$ and 0.161$^\circ$, respectively. And the longest measurable distance is 2525.3 mm and 3911.7 mm, respectively.
6. Conclusion
A novel panoramic stereo imaging system with triple-channel is proposed. It has a large FoV $360^{\circ }{\times }(0^{\circ }-105^{\circ })$ and a stereo imaging FoV $360^{\circ }{\times }(20^{\circ }-50^{\circ })$. The F number of the system is about 4.7 and the $F-\theta$ distortion is less than 10% in all FoVs. The system has a compact structure and provides a novel way to achieve stereovision with PAL. The total length is less than 81.114 mm and it has a simpler head unit. Besides, it has no central blind area, which gives it a great advantage compared to the other panoramic stereo imaging systems. The stereo angle resolution is higher than 0.143 $^\circ$, which gives it a good ability to distinguish the depth information. Considering its compact structure, it can be used in intelligent robots, vehicle protection, and other fields demanding panoramic stereovision. And due to its light and compact structure, the system can be used in mobile digital devices, which gives it another great potential. With the rapid development of information transmission technology, the amount of 3-D information obtained by the panoramic stereo system with high resolution can be delivered and processed fast. Since the low distortion and high resolution on the marginal FoV, it can be seen that the PAL system will be applied in machine visualization, intelligent environmental awareness, miniaturized robots, and many other fields.
Acknowledgment
This work is supported by the State Key Laboratory of Modern Optical Instrumentation of Zhejiang University.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. S. Gao, K. Yang, H. Shi, K. Wang, and J. Bai, “Review on panoramic imaging and its applications in scene understanding,” IEEE Trans. Instrum. Meas. 71, 1–34 (2022). [CrossRef]
2. Y. Huang, Z. Liu, Y. Fu, and H. Zhang, “Design of a compact two-channel panoramic optical system,” Opt. Express 25(22), 27691–27705 (2017). [CrossRef]
3. S. Gao, E. A. Tsyganok, and X. Xu, “Design of a compact dual-channel panoramic annular lens with a large aperture and high resolution,” Appl. Opt. 60(11), 3094–3102 (2021). [CrossRef]
4. S. Gao, L. Sun, Q. Jiang, H. Shi, J. Wang, K. Wang, and J. Bai, “Compact and lightweight panoramic annular lens for computer vision tasks,” Opt. Express 30(17), 29940–29956 (2022). [CrossRef]
5. X. Wang and Q. Xue, “Optical system design of an atmospheric detector with nadir view and omnidirectional limb view,” Appl. Opt. 56(26), 7454–7461 (2017). [CrossRef]
6. K. Yang, X. Hu, L. M. Bergasa, E. Romera, and K. Wang, “Pass: Panoramic annular semantic segmentation,” IEEE Trans. Intell. Transport. Syst. 21(10), 4171–4185 (2020). [CrossRef]
7. J. S. Chahl and M. V. Srinivasan, “Reflective surfaces for panoramic imaging: errata,” Appl. Opt. 38(7), 1196 (1999). [CrossRef]
8. T. Ma, J. Yu, P. Liang, and C. Wang, “Design of a freeform varifocal panoramic optical system with specified annular center of field of view,” Opt. Express 19(5), 3843–3853 (2011). [CrossRef]
9. H. C. Huang and Y.-P. Hung, “Panoramic stereo imaging system with automatic disparity warping and seaming,” Graph. Model. Image Process. 60(3), 196–208 (1998). [CrossRef]
10. W. Li and Y. F. Li, “Single-camera panoramic stereo imaging system with a fisheye lens and a convex mirror,” Opt. Express 19(7), 5855–5867 (2011). [CrossRef]
11. J. Wang, J. Bai, K. Wang, and S. Gao, “Design of stereo imaging system with a panoramic annular lens and a convex mirror,” Opt. Express 30(11), 19017–19029 (2022). [CrossRef]
12. Z. Huang, J. Bai, and X. Y. Hou, “Design of panoramic stereo imaging with single optical system,” Opt. Express 20(6), 6085–6096 (2012). [CrossRef]
13. Y. Luo, J. Bai, X. Zhou, X. Huang, Q. Liu, and Y. Yao, “Non-blind area pal system design based on dichroic filter,” Opt. Express 24(5), 4913–4923 (2016). [CrossRef]
14. Y. Luo, X. Huang, J. Bai, and R. Liang, “Compact polarization-based dual-view panoramic lens,” Appl. Opt. 56(22), 6283–6287 (2017). [CrossRef]
15. P. Greguss, “Panoramic imaging block for three-dimensional space,” (1986). US Patent 4,566,763.
16. D. R. Matthys, J. A. Gilbert, and P. Greguss, “Endoscopic measurement using radial metrology with digital correlation,” Opt. Eng. 30(10), 1455–1460 (1991). [CrossRef]
17. Optics Edmund, “Datasheet of Broadband Polarizing Plate Beamsplitters,” https://www.edmundoptics.com.
18. Z. Huang, J. Bai, T. X. Lu, and X. Y. Hou, “Stray light analysis and suppression of panoramic annular lens,” Opt. Express 21(9), 10810–10820 (2013). [CrossRef]
19. S. Niu, J. Bai, X. yun Hou, and G. guang Yang, “Design of a panoramic annular lens with a long focal length,” Appl. Opt. 46(32), 7850–7857 (2007). [CrossRef]
20. K. Zhang, X. Zhong, L. Zhang, and T. Zhang, “Design of a panoramic annular lens with ultrawide angle and small blind area,” Appl. Opt. 59(19), 5737–5744 (2020). [CrossRef]
21. Z. Zhang, Y. Huang, S. Bao, and Z. Liu, “Panoramic annular image restoration algorithm by prediction based on the lens design characteristics,” Appl. Opt. 62(3), 518–527 (2023). [CrossRef]
22. X. Wang, X. Bu, B. Yang, G. Li, and J. Yu, “Corrected unwrapping method based on the tangential and radial distortion centre for the panoramic annular image,” IET Image Process. 9(2), 127–133 (2015). [CrossRef]
23. Q. Zhou, Y. Tian, J. Wang, and M. Xu, “Design and implementation of a high-performance panoramic annular lens,” Appl. Opt. 59(36), 11246–11252 (2020). [CrossRef]
24. J. Wang, Y. Liang, and M. Xu, “Design of panoramic lens based on ogive and aspheric surface,” Opt. Express 23(15), 19489–19499 (2015). [CrossRef]
