1. Introduction

Imaging 3D objects underwater is of great importance in a variety of tasks such as underwater environment exploration [1,2], biological protection [3,4] and robot navigation [5–8]. However, because of the instability of the water flow and floating in the water, setting multi-sensor-based systems, like camera array, multi-camera and depth camera, etc., are not applicable in real-time 3D imaging underwater. Plenoptic camera [9], which can capture 4D spatial and angular information via a single shot by inserting a microlens array (MLA) between the main lens and imaging sensor, shows high potential in these application areas because of its compact and stable architecture and high imaging speed adapting to the water flow.

The underwater application of a plenoptic camera necessitates the incorporation of a waterproof shell positioned in front of the lens. Compared to a flat shell, which reduces the field of view (FOV), a spherical shell presents the potential to preserve the FOV by appropriately positioning the inner camera [10,11], which is utilized in our underwater plenoptic camera system. Nevertheless, the underwater environment poses challenges, including absorption, scattering, and refraction, which can degrade the imaging performance of the plenoptic camera that has been optimized for air conditions. While the effects of absorption and scattering are typically alleviated through postprocessing techniques [12,13], it becomes essential to optimize the underwater plenoptic camera for water refraction. As shown in Fig. 1, after adding a spherical shell, two times of refraction are added during the ray propagation process, and the geometric parameters of the spherical shell will lead to imaging property variations in image clarity. In addition, the difference in the refractive index of air and water will also cause a difference in the ray refraction process, further causing changes in imaging characteristics such as FOV and image clarity. Therefore, it is necessary to optimize the physical parameters such as the distance between the spherical shell and the main lens, and the distance between the main lens and the MLA during the design and assembly process of the underwater plenoptic camera.

 

Fig. 1. The configuration of: (a) the focused plenoptic camera and corresponding subaperture image crop; and (b) the underwater focused plenoptic camera adding a spherical shell and its corresponding subaperture image crop.

Download Full Size | PDF

To the best of our knowledge, there lacks scientific research addressing the issue of imaging quality degradation caused by the water and the waterproof shell during the design and optimization of underwater plenoptic cameras. However, there exists some work on either optimizing plenoptic cameras in the air or designing cameras for underwater use. They can be divided into two groups: analysis or optimization of plenoptic cameras in the air [14–17] and design of underwater cameras [18,19]. To analyze or optimize a plenoptic camera in the air, C. Perwaß et al. [14] described the relationship between its optical parameters (such as object distance, image distance, and distance between MLA and sensor) and performance such as angular resolution, depth of field, etc., providing an initial direction for camera design. V. Drazic et al. [15], T. Li et al. [16] and T. Michels et al. [17] proposed improved plenoptic camera models with better depth discrimination, image resolution or depth of field, which provide valuable insights for optimizing plenoptic cameras in the air but do not solve the problem of image clarity degradation and FOV change when using the plenoptic camera underwater. For the design of underwater cameras, R. Qu et al. [18] have developed an underwater optical system that meets the required FOV but their method cannot be directly applied to optimize underwater plenoptic cameras due to the difference in structure. Guatek company [19] utilized the Raytrix R5 to design an underwater plenoptic camera with a flat shell, which reduces the FOV to 1/1.33 compared to that in air, with limited optimization. Moreover, the design method is not suitable for optimizing the underwater plenoptic camera with a spherical shell. In summary, although existing works offer some strategies for optimizing plenoptic cameras in the air or designing cameras for underwater use, none of them has successfully designed and optimized high-performance underwater plenoptic cameras with a spherical shell while maintaining the imaging performance. Therefore, an optimization method that considers both image clarity and FOV is highly desired for underwater plenoptic cameras.

This paper proposes a ray propagation model and physical parameter optimization method for the underwater plenoptic camera. First, after adding a spherical shell in front of the main lens, a geometric optics model for the plenoptic camera with a spherical shell is conducted to quantify the object-image relationship, based on ray propagation analysis. Secondly, after the calibration of the minimum distance between the spherical shell and the main lens, an optimization model for the underwater plenoptic camera is proposed to reduce the impact of the spherical shell and water propagation, while ensuring successful assembly. This optimization method effectively improves the performance in FOV of the underwater plenoptic camera and the image quality captured by the underwater plenoptic camera. The correctness and robustness of the proposed model to keep image clarity and FOV are verified by simulation, and the effectiveness of the proposed model is further demonstrated by designing and testing a real underwater plenoptic camera in the underwater.

The rest of this paper is organized as follows. Section 2 describes the proposed ray propagation model and optimization method for the underwater plenoptic camera. Experimental results are provided in Section 3 followed by conclusions in Section 4.

2. Ray propagation model and optimization method proposed for underwater plenoptic cameras

Unlike traditional plenoptic cameras, underwater plenoptic cameras are required to image objects in water, rather than in air. Furthermore, a waterproof spherical shell with the functionality of water resistance and optical transmission is added before the main lens of plenoptic cameras to protect the inner system from being soaked in water and transmit the rays from the water to the inner plenoptic camera. These new features sacrifice the imaging performance unless physical parameters are optimized, such as optimizing the distance between the shell and the main lens, optimizing the distance between the main lens and the MLA, etc. To compensate for the loss of imaging performance, an optimization model for those distances is proposed. Ray propagation analysis for the spherical shell is proposed in this section by analyzing its unique optical properties in the water, thus giving an overall underwater plenoptic camera geometric optics model. By combining the geometric optics model with plenoptic theory, the clarity optimization model is derived first to improve image clarity. Moreover, to keep the FOV of the underwater plenoptic camera and ensure successful assembly, a joint optimization model is then proposed to optimize image clarity and FOV simultaneously after the calibration for the minimum distance between the spherical shell and main lens in the underwater plenoptic camera.

2.1 Geometric optics model for the spherical shell of underwater plenoptic cameras

To analyze the ray propagation process in the underwater plenoptic camera, a geometric optics model for the spherical shell is proposed here. Without loss of generality and simplicity, the shell can be modeled as a two-layer monocentric spherical refraction system, which can be analyzed using a two-stage refraction process. Using Galilean-mode [9,20] underwater plenoptic camera with a spherical shell as an instance, the optical structure is shown in Fig. 2, based on which the model is derived. For Kepler-mode [9,20] underwater plenoptic camera, the following derivations can be directly transferred by placing the immediate image plane in front of the image plane, which also means replacing ${a}$ with $- {a}$.

 

Fig. 2. Ray propagation process of Galileo-mode underwater plenoptic camera.

Download Full Size | PDF

As shown in Fig. 2, when considering the spherical shell, the convergence of rays originating from the paraxial object point A may not result in a perfect focal point due to spherical aberration. However, it is still possible to approximate the formation of a small image point using Gaussian optics [21] and paraxial ray analysis [22], as follows:

Equation (1) assumes that object point A is located near the optical axis and introduces more errors as the distance between point A and the optical axis increases. This equation is commonly used to calculate the focal length of a convex or concave lens [23,24], under the assumption of a distant object (${d_{OBJ\_A}} = \textrm{infinity}$), where the distance ${l_{OA^{\prime}}}$ becomes the focal length. To keep near-axis objects in focus, the following two-stage refraction process is analyzed based on Eq. (1)

In stage 1, paraxial object point A is imaged by the shell’s outer surface to point $A^{\prime}$, the refraction process follows:

In stage 2, the inner spherical shell shutters the ray transferred from the first refraction. The rays can be traced back to produce a virtual point $A^{\prime\prime}$. The position of point $A^{\prime\prime}$ follows:

Adding the condition in Fig. 2 that ${l_{O^{\prime}A^{\prime}}} = {l_{OA^{\prime}}} - {l_{O^{\prime}O}}$, substituting Eq. (2) into Eq. (3) and replacing ${l_{O^{\prime}A^{\prime}}}$ by ${l_{OA^{\prime}}} - {l_{O^{\prime}O}}$, gives a relationship between ${l_{O^{\prime}A^{\prime\prime}}}$ and ${d_{OBJ\_A}}$ by

Then the distance between virtual point $A^{\prime\prime}$ and the main lens, i.e., the equivalent object distance for the main lens, is given by:

As shown in Eq. (5), given the fixed geometric parameters of the spherical shell, the equivalent object ${l_{A^{\prime\prime}M}}$ is a function of ${l_{O^{\prime}A^{\prime\prime}}}$ and e, in which ${l_{O^{\prime}A^{\prime\prime}}}$ is a function of ${d_{OBJ\_A}}$ defined in Eq. (4). After modeling the paraxial ray propagation process of a spherical shell, it is possible to judge if the object point is in focus. In other words, if we can design an underwater plenoptic camera that follows Eq. (5), we can make sure that the camera can capture clear plenoptic images in the object distance we set.

2.2 Optimization method for the underwater plenoptic camera

Based on the derivations in Eq. (5), the difference between the equivalent object distance ${l_{A^{\prime\prime}M}}$ and ${d_{OBJ\_A}}$ is found, which explains the reason why the central perspective image in Fig. 1(b) becomes blurred when adding a spherical shell directly. So, the clarity optimization model is derived first to reduce blurry effects. Then the minimized distance between the spherical shell and the main lens is obtained by calibration, based on which a joint optimization model is proposed to optimize image clarity and FOV at the same time.

Reducing the blurry effects underwater is to make the camera focus on the object. According to plenoptic imaging theory [9,14], if the object distance changed from ${d_{OBJ\_A}}$ to ${l_{A^{\prime\prime}M}}$, the image distance v, the distance between the MLA and the image plane, b, and the distance between the image point of the main lens and the MLA, a, should be changed. However, distance a and b also need to follow [25]:

Equation (8) provides an implicit expression of $\Delta v$ related to ${l_{A^{\prime\prime}M}}$, which is a function of ${d_{OBJ\_A}}$ and e. Using this equation, $\Delta v$ can be optimized as a variable while fixing the object distance ${d_{OBJ\_A}}$ and distance e, which leads to the focus of the underwater plenoptic camera and improvement of image clarity. The optimization that satisfies Eq. (8) is also called clarity optimization in the following part.

During the clarity optimization process, variations in the parameter e can lead to a set of solution pairs for $\Delta v$ and e. To keep the FOV of the underwater plenoptic camera as close as the inner plenoptic camera, the distance e should be as small as possible [10]. In the actual assembly of the underwater plenoptic camera, if the distance e between the principal point of the main lens and the spherical shell is set to 0 mm, adjusting the main lens to the center of the spherical shell can help minimize refraction effects [27]. This alignment ensures that the field of view (FOV) remains consistent between the main lens and the inner camera, as depicted in Fig. 3 (b).

 

Fig. 3. The relationship between FOV of the inner camera and underwater camera for different e.

Download Full Size | PDF

In addition to the case mentioned above where $e = 0$, there are two more situations where the distance e can vary. In Fig. 3(a), when the distance e is less than 0, the field of view (FOV) of the underwater camera ${\alpha _{WATER}}$ is larger than that of the inner camera $\alpha $ while keeping the FOV of the inner camera fixed. When the principal point of the inner camera is located right on the inner wall of the spherical shell ($e ={-} {l_{O^{\prime}C}}$), the situation becomes similar to a flat shell, as the curvature of the spherical shell appears to have a radius of infinity for the inner camera.

On the other hand, in Fig. 3(c), as the distance e increases beyond 0, the FOV ${\alpha _{WATER}}$ of the underwater camera becomes smaller than the FOV $\alpha $ of the inner camera. However, the distance e cannot exceed ${l_{O^{\prime}C}}/\textrm{tan}({\mathrm{\alpha }/2} )$ to maintain the inner camera's ability to observe the surrounding environment through the spherical shell, thereby preventing the occurrence of black edges in the captured images.

In practice, the distance e is constrained because of the nontrivial distance between the front surface of the main lens and its principal plane. Figure 4 shows the diagram of the underwater plenoptic camera with a commercial lens. In this configuration, the commercial lens is positioned closest to the spherical shell so ${l_{bound}}$ represents the minimum distance between the center of the spherical shell, C, and the optical center of the commercial lens, $C^{\prime}$. Consequently, the distance e must satisfy $e \ge {l_{bound}}$, in case of assembly. For a given commercial lens, the exact position of its optical center is difficult to retrieve as lacking detailed information on the optical structure. So, calibration for ${l_{bound}}$ is conducted as follows.

 

Fig. 4. The diagram of the underwater plenoptic camera with a commercial lens.

Download Full Size | PDF

First, we employ our calibration method for the inner plenoptic cameras [28,29]. This method utilizes a triple-level calibration board consisting of three separate boards with known depth disparity as the calibration object. By analyzing different optical parallaxes at various depths, we derive the geometric parameters, specifically the distance between the principal plane of the main lens and the principal plane of the MLA, denoted as d, as well as the distance between MLA and sensor, denoted as b. Secondly, as shown in Fig. 4, for a paraxial calibration object point B, suppose the image points of B in adjacent microlenses are ${P_0}$ and ${P_1}$, and the distance between the two points is ${l_{{P_0}{P_1}}}$, object distance of MLA ${a_B}$ can be derived based on similar triangle principle [30] by substituting b into:

Since the equivalent object distance of object point B, i.e., ${l_{B^{\prime\prime}C^{\prime}}}$, follows Gauss equation:

As a result, to combine the condition $e \ge {l_{bound}}$ and keep the FOV of the underwater plenoptic camera close to that of the inner plenoptic camera, the distance e should follow:

The optimization that satisfies Eq. (13) is also called FOV optimization. Thus, the model satisfying Eq. (5), Eq. (8) and Eq. (13) improves image clarity and optimizes FOV simultaneously, which is so-called joint optimization in the following part.

In addition, when the image distance is modified from v to $v + \Delta v$, the aperture size of the main lens needs to be adjusted accordingly to maximize the utilization of the image sensor [14], which can be expressed as follows:

Figure 5 summarizes the process of optimizing image clarity and FOV: Using a plenoptic camera and a spherical shell, an underwater plenoptic camera is built with initial values of $\Delta v$ and e. Then, from the geometric parameters of the spherical shell, ray propagation analysis of the spherical shell using Eq. (5) is derived to obtain expressions of equivalent object distance for the main lens. Following ray propagation analysis, clarity optimization uses Eq. (5) and Eq. (8) to obtain a set of solution pairs for $\Delta v$ and e due to the variations of e, which leads to the improvement of image clarity. Next, through calibration, the minimum distance between the optical center of the commercial main lens and the center of the spherical shell, ${l_{bound}}$, is obtained. Then, the FOV optimization model, using Eq. (13) and the calibrated value of ${l_{bound}}$ to find optimized e, is executed to find a unique solution for $\Delta v$ and e. The joint optimization model, which combines clarity optimization and FOV optimization, uses Eq. (5), Eq. (8) and Eq. (13) to optimize clarity and FOV simultaneously. Then after adjusting the aperture size of the main lens using Eq. (14), optimized parameters of underwater plenoptic camera are designed.

 

Fig. 5. The process of optimizing an underwater plenoptic camera.

Download Full Size | PDF

It is important to note that during the optimization of an underwater plenoptic camera, certain parameters such as the radius and refractive index of the spherical shell, refractive index of water, and object distance are considered as prior information. In the event of changing circumstances, these parameters can be adjusted and utilized for re-optimizing the underwater plenoptic camera using the same optimized procedures shown in Fig. 5. This may involve directly modifying these parameters, such as adjusting the radius and refractive index of the spherical shell, or taking into account hidden variables that can influence these parameters, such as the effects of temperature [31] or electrolyte concentration [32] (e.g., salt content) of the seawater on the refractive index of water.

After computing the optimized distances $\Delta v$ and e, an adjustment strategy is employed during the actual assembly process as outlined below. Firstly, $\Delta v$ is adjusted in air condition. By utilizing our calibration method [28,29], we can calibrate the image distance of the main lens ${v_{ADJ\_1}}$ after the initial adjustment and compare it with the image distance v, then we calculate the difference before and after the initial adjustment, which can be expressed:

Next, we proceed to adjust the distance e for the underwater plenoptic camera after incorporating the spherical shell. To achieve this, the plenoptic camera’s outer radius is designed to match the inner radius of the waterproof housing. By ensuring that the optical axis of the plenoptic camera aligns with the middle of the housing, we can guarantee that their optical axes remain in the same line [33]. Consequently, the distance e can be adjusted in a one-dimensional manner to regulate the separation between the plenoptic camera and the spherical shell, thereby ensuring that the image points of the object at adjacent microlens distances ${l_{{P_0}{P_1}}}$ in Fig. 4 adhere to the following relationship, which serves as an indication of the plenoptic camera being in focus:

2.3 DoF analysis of the optimized underwater plenoptic cameras

The underwater plenoptic camera can capture underwater angular and spatial information with extended depth of field, allowing for certain variations in object distance resulting from object movement or positional errors. However, when substantial changes in object distance occur, it becomes essential to evaluate the necessity of re-optimizing the entire underwater plenoptic camera. To address this concern, we propose a depth of field (DoF) analysis for the optimized underwater plenoptic camera. The purpose of this analysis is to get depth of field of the underwater plenoptic camera and evaluate whether a re-optimization of the camera is necessary when there are significant changes in the object distance.

It is well known that a camera image is in focus within a certain range of distances from the lens. Beyond this range, the image becomes defocused and can be characterized by a blur circle. For the underwater plenoptic camera, the range of distances that appear in focus corresponds to a blur circle, denoted as $\delta ^{\prime}$, which satisfies the condition $\delta ^{\prime} \le \delta $, where $\delta $ represents the maximum acceptable blur diameter. Accordingly, the virtual object distance ${l_{D^{\prime\prime}M}}$ in Fig. 6 is constrained to make the plenoptic camera in focus, at the same time ${l_{D^{\prime\prime}M}}$ is a function of ${d_{OBJ\_D}}$ described by Eq. (5), which is shorter or longer than the virtual object distance that is set currently. Using the process to calculate the diameter of blur circle for shorter focused virtual distance ${l_{D^{\prime\prime}M}}$ as an instance, the diameter of blur circle ${\delta _D}$ is derived. It is noted that during the design of plenoptic cameras, the pixel size cannot be small [34] and diffraction effects are neglected in the following derivations.

 

Fig. 6. Relationship between object distance and blur circle.

Download Full Size | PDF

After using Eq. (5) to calculate the shorter focused virtual distance ${l_{D^{\prime\prime}M}}$, image distance of main lens ${v_D}$ can be obtained as follows:

Using ${v_D}$ from Eq. (17) and calibration value d, object distance of MLA ${a_D}$ can be expressed as ${a_D} = {v_D} - d$, so image distance ${b_D}$ for MLA can be calculated by Gaussian equation:

Based on similar triangle principle, relationships between ${\delta _D}$ and ${b_D}$ can be given by:

Other than ${d_{OBJ\_D}}$, Eq. (17), Eq. (18) and Eq. (19) can also be utilized to analyze the tolerance for installation errors since the relationship between ${l_{D^{\prime\prime}M}}$ and installation parameter e is already given by Eq. (5).

3. Experimental results

In this section, the correctness and effectiveness of the proposed optimization method are demonstrated by testing on the simulated system and the real imaging system.

3.1 Test on the simulated system

To verify the accuracy of our calibration method, standard optical simulation software [35] is used to simulate the imaging results of the plenoptic camera and underwater plenoptic camera with square-arranged MLA. To verify the effectiveness of the proposed optimization model in improving the image clarity and FOV, we first compared four cases corresponding to four stages in Fig. 5 using simulation: Plenoptic Camera, which corresponds to the input plenoptic camera in Fig. 5 without the spherical shell serving as the reference; Adding a Spherical Shell Directly, which corresponds to the step in Fig. 5 to build an underwater plenoptic camera without any optimization; Clarity Optimization represents optimizing the underwater plenoptic camera using Eq. (5) and Eq. (8); Joint Optimization represents optimizing the underwater plenoptic camera using Eq. (5), Eq. (8) and Eq. (13).

The reference case Plenoptic Camera is a focused plenoptic camera with an ideal lens capable of capturing 3D information [14], whose angular resolution equals to 3.13 and depth resolution equals to 30 mm, given 800 mm as object distance ${d_{OBJ}}$ and 1/2 pixel as minimum distinguishable parallax change. The values of its optical parameters are listed in Table 1, where the position of the main lens represents the principal point of the main lens.

Table 1. Optical Parameters Values of Plenoptic Camera

View Table | View all tables in this article

Table 2 lists the values of optimized variables e and $\Delta v$ in the other 3 testing cases. As shown in the table, in the case of Adding a Spherical Shell Directly, optimized variable e is set as 30 mm and $\Delta v$ is 0 mm, which is also the initial value of subsequent optimization. After Clarity Optimization, optimized variable e remains 30 mm and $\Delta v$ is derived as 1.745 mm. Then, setting ${l_{bound}}$ to 0 mm in the simulation and using Joint Optimization, the optimized variables e and $\Delta v$ are derived as 0 mm and 2.205 mm, respectively. It is worth mentioning that while we have specifically set 30 mm as one of the distance values for e, it serves as a representative and effective point of comparison. Our findings indicate that other distances yield similar results, further reinforcing the validity of our observations.

Table 2. Values of Optimized Variables in the other 3 Testing Cases

View Table | View all tables in this article

Image clarity and FOV of underwater plenoptic cameras in the other 3 testing cases are compared by simulation using corresponding values of optimization variables in Table 2. Three objects with representative textures are simulated and compared, within which the ChArUco calibration board with regular grids and sharp edges provides a striking contrast, two fishes with sharp and colorful textures demonstrate the robustness of the method. Four-step operations including simulation, rendering, image processing and evaluation are used to quantitatively evaluate the image clarity of simulated underwater plenoptic images. Using the ChArUco calibration board in Fig. 7(a) as an instance, first, it is used as an object for simulating the plenoptic images. Secondly, a rendering algorithm based on appropriate patches [28] is used to obtain central subaperture images, which uses SSIM [36] as an evaluation to pick the appropriate patch size as shown in Fig. 7(b) and stitches extracted patch to produce the rendered subaperture images as shown in Fig. 7(c). Then, pixel value ranges of each subaperture image are normalized to [0, 255] so that the comparisons are not disturbed by light transmission efficiency. After the above operations, the Energy of Gradient (EOG) [37,38] is chosen as a criterion to evaluate the clarity of images due to its simplicity and accuracy. Let $f({x,\; y} )$ represent the value of pixel $({x,y} )$, EOG can be calculated as:

 

Fig. 7. (a) ChArUco calibration board; (b) best-matched patch and corresponding calculation method of patch size; (c) rendered subaperture images captured by a simulated plenoptic camera before adding a spherical shell in air condition.

Download Full Size | PDF

A larger EOG means that the corresponding image has higher clarity than a smaller EOG. Besides, subjective evaluation is used to compare FOVs, the difference between which are significant in different situations. All tested subaperture images and their corresponding EOGs are shown in the even rows of Fig. 8.

 

Fig. 8. Simulated plenoptic images and corresponding central subaperture images of: (a) and (b) ChArUco calibration board; (c) and (d) guppy; (e) and (f) Oscar fish under different testing cases.

Download Full Size | PDF

Figure 8 (a)-(b) shows the simulated plenoptic images of the ChArUco calibration board and corresponding central subaperture images. As shown in the figure, the EOG of the subaperture images simulated using parameters with Clarity Optimization is 5380, which is 2.34 times higher than that of the subaperture images simulated using parameters with Adding Spherical Shell Directly, and present a less blurry effect, which demonstrates the effectiveness of Clarity Optimization. Moreover, the EOG of the subaperture images simulated using parameters with Joint Optimization is 6200, which is much higher than that of Clarity Optimization, at the same time FOV of underwater plenoptic images with Joint Optimization stays much closer to the FOV of the Plenoptic Camera used in the air than that of other two situations, which substantiates the effectiveness of our method in improving image clarity and optimizing FOV.

Figure 8 (c)-(f) shows the simulated plenoptic images of the fishes and corresponding central subaperture images. As shown in the figure, the EOGs of underwater plenoptic images with Clarity Optimization and Joint Optimization are 2.52 times and 5.44 times that of simulated central subaperture images using parameters with Adding Spherical Shell Directly, and at the same time, the subaperture images simulated using parameters with Joint Optimization shows less blur and keeps more consistent FOV with Plenoptic Camera used in air under subjective evaluation, which shows the robustness of our method in optimizing FOV and image clarity with characteristics of underwater life.

3.2 Test on a real imaging system

To further demonstrate the effectiveness and robustness of our optimization model, the optimization model is applied to a real self-designed focused plenoptic camera [9] in Fig. 9(a). Our calibration method [28,29] is used to obtain the physical parameters of the plenoptic camera, which are listed in Table 3. Like simulation, the focused plenoptic camera can be used to perform 3D reconstruction in the air because its angular resolution is 3.26 and its depth resolution is 36 mm while setting the object distance to be 800 mm and using 1/2 pixel as the minimum distinguishable parallax change [14].

 

Fig. 9. Real imaging system. (a) Inner self-designed focused plenoptic camera; (b) self-designed underwater focused plenoptic camera; (c) configurations of underwater experiments.

Download Full Size | PDF

Table 3. Optical Parameters Values of Self-designed Focused Plenoptic Camera

View Table | View all tables in this article

In real experiments, the quantitative shift of e is not easy, so we only provide the comparisons between parameters of the cases Adding Spherical Shell Directly and Joint Optimization, which is enough to further prove the effectiveness of our proposed optimization model. The models Adding Spherical Shell Directly and Joint Optimization are consistent with the simulation experiments. In real experiments, ${l_{bound}}$ is calibrated to be 19 mm using the proposed calibration method in Eq. (12), corresponding values of optimized variables, including which set in the case Adding Spherical Shell Directly and derived using Joint Optimization, are listed in Table 4. The assembled underwater plenoptic camera with an acrylic spherical shell is shown in Fig. 9(b).

Table 4. Values of Optimized Variables in 2 Test Cases

View Table | View all tables in this article

Using the values of the optimized variables in Table 4, underwater plenoptic cameras in 2 test cases are compared. Four typical real underwater scenes including coral, fishes and starfishes are established to test the actual performance of our self-designed underwater plenoptic cameras, we place these objects in an object distance of around 800 mm. An example of real scene configuration is shown in Fig. 9(c). Like the simulation, the results are also quantitively evaluated by comparing the image clarity and richness of textures, and the evaluation process of the real imaging system is slightly different from that of the simulation in the third step, to eliminate the difference in light transmission efficiency, we preprocess the subaperture images of the underwater plenoptic camera without optimization to keep their luminance consistent with the subaperture images of the optimized underwater plenoptic camera. Since the FOV optimization has been tested effectively in the simulation and there is no reference in real experiments, FOV is not compared in real experiments.

Real experiment results are shown in Fig. 10. In the figure, the EOGs of central subaperture images captured by the underwater plenoptic cameras Adding Spherical Shell Directly and with Joint Optimization are displayed below the corresponding subaperture images. From the calculated EOGs, the EOGs of subaperture images captured by the underwater plenoptic camera with Joint Optimization are all around 3 times that captured by the underwater plenoptic camera Adding Spherical Shell Directly. In addition, as shown in the figure, the subaperture images captured by the optimized plenoptic camera present a less blurry effect and more delicate textures. The results of EOG and subjective evaluation indicate that our optimization method is very effective and robust for underwater focused plenoptic cameras, which greatly benefits the underwater applications of focused plenoptic cameras.

 

Fig. 10. Real experiment results. Underwater plenoptic images and corresponding central subaperture images of: (a) starfish; (b) fish; (c) crucian; and (d) coral.

Download Full Size | PDF

3.3 Refocusing experiment on underwater plenoptic images

To evaluate the refocusing capabilities of our optimized underwater plenoptic cameras, a real underwater scene is created, consisting of two fish and grass. These fish are positioned at different object distances, with one fish placed around 500 mm as the foreground element in the upper half of the image, while the other fish and grass are placed around 900 mm as background elements in the lower half of the image. Figure 11 (a) illustrates an example of an underwater plenoptic image captured in this setup.

 

Fig. 11. Underwater plenoptic images with multiple depths and corresponding depth map, rendering results.

Download Full Size | PDF

In our study, two approaches for obtaining depth maps from plenoptic images are considered: direct estimation from original plenoptic images [9] and estimation from rendered multiview images [39,40]. We specifically employ the original plenoptic images to extract depth maps and enable refocusing for the underwater plenoptic images. The procedure for extracting depth maps and performing refocusing remains consistent with the methodology described in [9]. Thus, the resulting depth maps and refocusing images correspond directly to the virtual object distance of the inner plenoptic camera, disregarding aberrations, rather than the actual object distance that can be calculated as a function of the virtual object distance of the inner plenoptic camera.

Similar to Fig. 7, we employ the structural similarity index (SSIM) as a criterion to generate an array of patch size values for each microlens image, which also represents the depth map of the plenoptic image depicted in Fig. 11(b). The resolution of the depth map is consistent with the number of microlenses in the microlens array (MLA), which is relatively small due to the limited number of microlenses in our specific MLA design.

Considering that the refocusing capability of a plenoptic 2.0 camera is dependent on the selection of the pitch size [9], we utilize small and large patches to render subaperture images, as depicted in Fig. 11(c) and Fig. 11(d). From the figure, it is evident that when a small patch is employed, the foreground object (located in the upper half of the image) appears clear, while the background objects (such as grass and fish in the lower half) exhibit block artifacts. Conversely, when a large patch is used, the results are reversed, with the foreground object exhibiting block artifacts and the background objects appearing clear.

By incorporating the depth map in Fig. 11(b) during the rendering process, we can obtain artifact-free rendering aperture images of the scene, as depicted in Fig. 11(e). These images represent an “all-in-focus” view of the scene, where both foreground and background objects are captured with optimal clarity and fewer block artifacts.

4. Conclusion

Underwater plenoptic cameras can be used for a variety of underwater tasks, such as underwater 3D reconstruction and underwater inspection. In this paper, an optimization model is proposed to reduce the degeneration problems during the design of the underwater plenoptic camera. By analyzing the imaging process of the underwater plenoptic camera and the unique optical properties of the spherical shell in water, a paraxial ray propagation analysis for the spherical shell is proposed, and the overall geometric optics model of the underwater plenoptic camera is given. An optimization model is then proposed to compensate for the change in image clarity and FOV caused by the spherical shell and water while guaranteeing a successful assembly. Simulated and real system results show that our proposed method can effectively compensate for the performance degradation caused for the spherical shell and water, which is beneficial for the underwater use of plenoptic cameras. In the future, our work will focus on advancing the accuracy of depth map estimation obtained from rendered multiview images particularly by accounting for aberrations, investigating postprocessing techniques aimed at compensating for image blurring caused by wave-induced disturbances and using wave optics and Fourier optics to further improve the performance of underwater cameras.