Open Access
Add to BibSonomy
Abstract
For the enhancement process of underwater images taken in various water types, previous methods employ the simple image formation model, thus obtaining poor restoration results. Recently, a revised underwater image formation model (i.e., the Akkaynak-Treibitz model) has shown better robustness in underwater image restoration, but has drawn little attention due to its complexity. Herein, we develop a dehazing method utilizing the revised model, which depends on the scene depth map and a color correction method to eliminate color distortion. Specifically, we first design an underwater image depth estimation method to create the depth map. Subsequently, according to the depth value of each pixel, the backscatter is estimated and removed by the channel based on the revised model. Furthermore, we propose a color correction approach to adjust the global color distribution of the image automatically. Our method only uses a single underwater image as input to eliminate lightwave absorption and scattering influence. Compared with state-of-the-art methods, both subjective and objective experimental results show that our approach can be applied to various real-world underwater scenes and has better contrast and color.
© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Underwater images play an essential role in marine geological exploration, marine ecological protection, and marine resource development [1–3]. However, in contrast to imaging conditions on land, the underwater environment is complex and changeable, i.e., underwater imaging is affected by the water medium, which results in image degradation, for which there are several reasons. Firstly, the attenuation of light underwater is affected by the propagation distance and wavelength of light. As a result, the color of underwater images is usually dark and appears blueish or greenish. Secondly, the scattering effect of substances suspended in the water results in noise that causes low contrast and blurred details in underwater images [4–6]. Finally, the severe degradation of underwater images reduces their practical application value. Thereby studies on reconstructing the lost color of the underwater images are of great significance.
Different approaches have been devised to recover the true colors of underwater images. However, these methods have poor universality, and the color correction results prompt further improvement. To reconstruct the natural colors based on a single underwater image, we propose a technique that combines an underwater imaging model with improved accuracy to remove backscatter and enhance color saturation. Experiments on multiple underwater scenarios demonstrate that the proposed method can obtain better visual effects than existing approaches.
The main contributions of our study are summarized as follows:
- 1) We propose a dehazing method using a single underwater image as input based on a revised underwater image formation model [7–9] to improve the underwater image quality. Specifically, we design an imaging range estimation approach containing underwater image segmentation and smoothing to obtain an absolute depth map with relatively accurate depth information. Subsequently, we fit the functional relationship between the depth map and backscatter based on the revised model and select the critical area of relevant results to remove the backscatter of the original image.
- 2) We design an approach for an automated adjustment of the global color distribution of the image. Information entropy is used to automatically select the best global illumination parameter to improve the brightness, contrast, and color of the entire underwater image.
- 3) We verify the stability, practicality, and accuracy of the proposed method in both haze and real-world underwater scenarios.
2. Related works
This section introduces three imaging models: the atmospheric scattering model, the simplified underwater image formation model (IFM), the revised underwater image formation model, and discusses the corresponding methods relying on the aforementioned models. In addition, we will introduce some typical model-free enhancement methods, including traditional enhancement methods and methods based on convolution neural networks.
2.1 Imaging models and model-based underwater restoration methods
The images captured on foggy days are influenced by the absorption and scattering by the air medium, resulting in low contrast and dull colors. The model describing the image degradation process in fog is as follows [10]:
where x represents pixel coordinates, $I(x)$ represents the degraded image, $J(x)$ represents the scene radiance, $t(x)$ indicates the transmission map representing the portion of $J(x)$ reaching the camera, and A is the global atmospheric light. When the medium is uniformly distributed, $t(x)$ can be further expressed as: where $\beta$ denotes atmospheric attenuation coefficient, and $d(x)$ is the distance from the camera to the scene. Formula (2) shows that the transmission map decays exponentially with distance. In 2011, He et al. [11] proposed a haze removal method (Dark Channel Prior, DCP) based on the above model, which has provided a lot of inspiration for underwater image restoration [12–19].Compared to the atmospheric scattering model, the simplified underwater image formation model is more complicated owing to the selective absorption of light waves by water. In this aspect, the simplified underwater image formation model can be obtained as [12]:
Most of the restoration methods based on IFM have been inspired by DCP. Drew et al. [13] proposed the underwater dark channel prior (UDCP), which only uses green and blue channels to calculate the transmission map. Their method, however, is unsuitable for the images of the red channel without obvious attenuation. Galdran et al. [14] proposed the Red-Channel method, through which light of long wavelengths can be better restored. Peng et al. [15] generalized the DCP, but the subsequent results are unstable when restoring different types of underwater images. Zhou et al. [16] proposed a method where DCP was used to estimate the rough transmission map of underwater images with the inverted red channel. However, some results appear reddish, since the model reduces the influence of artificial light sources by increasing the value of the red channel.
Some scholars proposed other IFM-based approaches to solve the transmission map to obtain better results. Carlevaris-Bianco et al. [20] presented a method to eliminate the influence of scattering by using the different attenuation degrees of wavelengths underwater. The procedure runs without any hardware or prior knowledge, while its effect is not significant when the red light has not been strongly attenuated. Li et al. [21] proposed an approach based on minimizing information loss, including a restoration method based on IFM and an enhancement method based on histogram distribution. However, the technique causes red artifacts in images with high red channel attenuation. Peng et al. [22] introduced an approach based on blurriness and light absorption, but estimating blurriness maps is time-consuming, making it unsuitable for real-time applications. Berman et al. [23] designed an underwater image color restoration approach. Firstly, the transmission map of the degraded image under each water type is repeatedly estimated in this process. Next, the color of the image is restored based on IFM, and the best result is automatically selected. Therefore, this method requires a lot of time for repeated estimation, resulting in low efficiency.
Studies have shown that a simplified underwater imaging model is only a rough estimate of the underwater imaging process. It ignores the difference in the dependence of the direct attenuation coefficient and backscattered attenuation coefficient. It was proved that the attenuation coefficient of direct reflected light relies on the imaging range and the reflectance of the object, and the error caused by neglecting these correlations through in situ experiments in two optical water bodies was quantified [7]. Based on this work, Akkaynak et al. studied the functional relationship and parameter dependence among the components, and proposed a revised underwater imaging model [8]:
In 2019, Akkaynak and Treibitz [9] proposed a method (Sea-thru) based on the revising model. Although Sea-thru ignores part of the parameter dependence of the revised model, the results are still significantly improved. Due to the strong dependency between the revised model and depth information, however, Sea-thru requires RGB-D image as its input, making this method unfit for processing underwater images lacking depth information.
2.2 Model-free underwater image enhancement methods
Further methods have been presented that do not rely on a physical model, and improve image quality by directly adjusting the value of pixels. Ancuti et al. [24] proposed an approach that fuses different feature images into one image by setting weights. On this basis, they further designed a multi-scale fusion approach combining white balance [25], which can obtain better results on underwater images with severe light attenuation. Fu et al. [26] introduced a two-step enhancement approach, whereas Lee et al. [27] used an improved white balance algorithm and a normalization algorithm to enhance underwater images based on DCP. This hardly produces red artifacts, but superpixel-based restoration will result in an unsmooth processed image.
Due to its robust feature learning ability, deep learning has been widely used recently in various visual tasks [29,30]. Its application in underwater image enhancement has become a development trend [31–34,36]. Li et al. [33] designed an underwater image enhancement model based on CNN (UWCNN), which uses synthetic underwater images for training and an end-to-end approach to reconstruct clear underwater images directly. They synthesized ten ocean image datasets and trained ten corresponding UWCNN models corresponding to the optical properties of different water types. However, the appropriate water type cannot be automatically selected. In 2018, Li et al. [34] introduced an unsupervised color correction model based on GAN [35] and called it WaterGAN, which can generate a dataset containing color-corrected images and their depth information. In 2020, Li et al. [36] constructed a real underwater image dataset containing multiple scenes with their corresponding high-quality reference images, and proposed an underwater image enhancement network (Water-Net) based on the constructed dataset. Water-Net can achieve better results, although it is only a basic framework, and researchers can obtain superior results by changing the network structure and modifying the loss function.
3. Methods
This section introduces a simplified imaging model based on the revised model in the literature [8]. Secondly, we propose an approach for estimating the imaging range, then estimate and remove the backscatter of underwater images on this basis. Finally, we rebuild the brightness and color of underwater images using a simple and effective illumination estimation method.
3.1 Simplified model
The IFM has been widely used for underwater image restoration for many years. However, it lacks an accurate description of underwater imaging. On the one hand, IFM neglects the difference between the direct signal and the backscattered signal. On the other hand, this technique only considers the dependency of water properties while ignoring the dependence of scene properties. Due to its imprecision, the restoration results based on IFM are unreliable and unstable. Based on the above situation, Akkaynak et al. [8] proposed a more accurate model for underwater image restoration. However, due to its complexity, the new image formation model has attracted little attention. To use the revised model, inspired by Sea-thru [9], we only consider the effect of imaging range on backscatter. Regardless, the simplified revised model is still more accurate than IFM. The improved underwater imaging model described in the Refs. [7–9] is simplified to:
3.2 Imaging range estimation
The above model strongly depends on the imaging range of the scene, which means that an accurate depth map is essential for processing. Single-image depth estimation is a classical problem in computer vision, which has great importance for 3D scene reconstruction. However, traditional depth estimation methods based on MRF [37,38] or CRF [39] modeling has shown limited accuracy. Due to the powerful feature extraction capabilities of the neural network, deep learning approaches often present better robustness. However, the supervised depth estimation method is burdened by the difficulty of obtaining pixel-level depth datasets. In contrast, unsupervised depth estimation methods can obtain more accurate estimation results. Benefiting from three optimization strategies of [40], self-supervised monocular depth estimation method generates superior predictions. Firstly, a minimum reprojection loss strategy is proposed to avoid the problem that the projection position is correct while loss increases due to occlusions. Secondly, a multi-scale estimation strategy is used to reduce artifacts in depth maps. Finally, an auto-masking loss method is designed to avoid pixels that violate the camera motion assumptions from interfering with the training process. We combine our approach with the above-mentioned advantages of [40] to obtain an accurate depth map estimation.
Inspired by the Sea-thru method [9], we use the unsupervised monocular depth estimation method [40] to estimate the depth map of underwater images. However, due to the low contrast and serious color cast of underwater images, almost all results of [40] incorrectly estimate part of the underwater image background to a minimal value (close to the camera), resulting in a significant error in the step of estimating backscatter, as shown in Fig. 2(a). Therefore, we firstly use contrast stretching algorithm to process the original image, that is, adjust each pixel of the original image to between 0 and 255. Subsequently, we use [40] to estimate the depth map of the image after contrast stretching. After obtaining the depth map, we re-estimated the background area where the depth estimation was wrong. Finally, we use guided filtering [28] to smooth and enhance the details of the depth map.
Fig. 2. Diagram of the depth map estimation. The top raw is the original image and the original image after stretching the contrast. The second row is the visual display of the depth map, where the red area indicates the pixels closer to the camera, while the blue area represents the pixels farther from the camera. (a), (b) and (c) show the results of haze removal based on different depth maps. For the convenience of observation, we appropriately improved the brightness of the image. (a) and (b) are the error results estimated based on inaccurate depth maps, and (c) is the de-scattering result obtained based on the depth map optimized by our method.
The method of re-estimating the background area with the wrong depth value is discussed next. Firstly, we use image segmentation to separate erroneous estimation pixels from the depth map based on the contrast-stretched image [41]. Compare to the original underwater image, the color contrast of the contrast-stretched image has been significantly improved as depicted in Fig. 2(a) and (c) which are the contrast-stretched image and the original image, respectively. Images after contrast stretching are easier to segment, because the image is segmented according to the color similarity between the selected region and each pixel. Therefore, we arbitrarily select part of the background area where depth estimation is inaccurate for the contrast-stretched image as the reference color sample point. As shown in Fig. 2(b), the RGB vector ${V_m}$ represents the region’s expectation (the line segment whose origin points to the m point), and m is the center of the sample to measure the similarity of each point in the image. Herein, Mahalanobis distance is used to measure the similarity between pixels in RGB space. The Mahalanobis distance $D(z,m)$ between any point z in the RGB space and center point m of the sample is shown in Formula (10):
where C indicates the covariance matrix of the selected sample. All sample points on the diagonal of the matrix $C$ are selected, and T is equal to the standard deviation of its maximum. The arbitrary sample point z satisfying in the whole image is the background area selected by this method. Figure 3(a) ∼ (e) shows the segmentation results after three times of sampling in the background area of Fig. 3(a), which proves that the proposed method accurately separates the background area from the original image. Next, we re-estimate the depth of the region. Since the background region of the image is the farthest region from the camera in the whole image, we estimate the depth value of the sample points in the result region as the first 1% of the original depth map, sorted from small to large.Fig. 3. The diagram of color segmentation. (a) Underwater image after contrast stretching. (b) Schematic diagram in 3D coordinates. The x-axis, y-axis, z-axis indicates the red, green and blue component, respectively. The blue dot indicates the position of all pixels in the RGB space of figure (a), point m represents the center point of the ellipsoid, and all points contained in the interior and surface of ellipsoid with point m as the center are the points satisfying the conditions. (c) Original image [36]. (d) Display segmentation result by the binary image. (e) Display segmentation result by the original image.
Estimating a specific part of the background as a value will cause the area to be significantly different from the surrounding area, resulting in uneven backscatter removal. Therefore, we improve the accuracy of the depth map by the guided filter. Figure 2 shows the estimating depth maps of different methods, and the corresponding different de-scattering results are given based on each depth map. To visualize the results, we adjusted the brightness of the de-scattered image. Figure 2 demonstrates that the wrong depth estimation will cause obvious errors in the de-scattering results.
3.3 Backscatter estimation
To facilitate the fitting and ensure the accuracy of the fitting result, we uniformly convert the obtained depth value into a value within a specific range, namely, an absolute depth map. For each relative depth value x in the original depth map, we use a linear conversion method to convert it to an absolute depth value y. The specific conversion formula is as follows:
After obtaining an accurate absolute depth map, based on the Sea-thru model [9], the depth information of each pixel is used as a classification standard. Under different classification results, black pixels are searched for as the initial estimated value of backscatter. In the specific implementation of this method, each pixel in the original image is divided into several groups according to its depth value. Since too many pixels are meaningless for the estimation of backscatter, we use the following method to get the number of points:
where ${n_1} = 1000$ is set, ${n_2}$ represents the total number of pixels in each group, and $N$ represents the number of pixels we take in each group. We take the first N smallest RGB triples (black pixels) of each group as the initial estimate of backscatter. According to this approach, the red points in the second row in Fig. 4 are the black pixels selected. We fit these points and their corresponding depth values according to Formula (13) for sub-channel fitting. Figure 5 shows the fitting results of sample images.Fig. 4. Original underwater images and images were containing the selected black pixels. The first row in (a)-(f) are original underwater images [9,36], and the second row shows the corresponding images with the selected black pixels. We display the selected black pixels in red (255, 0, 0) and appropriately enlarge each red point for the convenience of observation. It can be seen that these pixels usually appear in the shadow zone and with the imaging range from near too far. It is proven that our imaging range estimation method and black pixel selection method are accurate.
Fig. 5. Fitting result chart. (a) Original images [23,36]. (b) Result of removing backscatter and adjusting brightness. (c) Fitting result of backscatter. The abscissa represents the depth, and the ordinate represents the pixel value. Red, green, and blue dots represent RGB pixel values obtained from different depths, respectively; black curves represent three RGB curves fitted according to different RGB values.
3.4 Illuminant estimate and color recovery
The above method only solves the problem of haze caused by scattering. When the backscatter is removed from the original image I, color distortion still exists due to the absorption of light by water. Therefore, a color recovery method is necessary after restoration. Herein, we apply information entropy to find the best illuminant parameter for color equalization automatically. The information entropy $H(\xi )$ is calculated as follows:
where i represents the gray level of the pixel, and ${p_i}$ represents the proportion of the pixel with the gray level i in the whole image. The larger the value of information entropy, the more information the image contains. In estimating illuminant, we sort the pixel values of three channels and take the pixel’s value from the top 0.5% to 2% at 0.15 intervals of each channel ${W_{{c_1}}},{W_{{c_2}}},{W_{{c_3}}}\ldots {W_{{c_n}}}$. The enhanced image ${D_{{r_i}}}$ can then be obtained by Formula (15): where ${D_c}$ represents the underwater image after backscatter removal. ${W_0}\textrm{ = 0}\textrm{.1}$ is set to ensure that the image will not be over-enhanced in a certain channel. The final result among ${D_{{r_1}}},{D_{{r_2}}},{D_{{r_3}}}\ldots {D_{{r_n}}}$ can be obtained by (16):4. Experimental results
Our method includes a dehazing method based on an underwater imaging model with improved accuracy, and an enhancement method that adjusts the overall chroma and saturation of the image. Therefore, the experiment is divided into two parts to verify the effectiveness of the proposed dehazing approach and the overall method, respectively.
4.1 Validation of image dehazing method
To verify the dehazing performance of our method, we select several representative methods, including the approaches of Tarel et al. [42], He et al. [11], Zhu et al. [43], Berman et al. [44], and Cai et al. [45] as the comparison methods, and experiment on both natural and synthetic haze images. Figure 6 illustrates the sources of each image in Figs. 7–8. Figure 7 presents the comparison results of ten hazy images. As depicted in the figure, the results of Tarel et al. [42] introduce serious noise, especially at the edge of the object in each image. This is because the filter proposed by Tarel et al. enhances edges and corners excessively. He’s [11] method has an obvious dehazing effect but still shows some inaccuracies (Fig. 7(a), (b)), since it features the inherent problem of overestimating the transmission map. The effects of [43], [45] are not obvious, while the method of Berman et al. [44] has significant dehazing results. Differently from the patch-based method of He et al. [11], the method proposed by Berman et al. [44] is a pixel-based method. Thereby it copes well with the image of various depth details. However, the algorithm may fail in the region covered by the sky (Fig. 7(h)). Compared with the above methods, our proposed method obtains dehazing results with moderate brightness and vivid color information.
Fig. 6. The sources of each raw image in Figs. 7–8. (a)-(j) are from RESIDE [46]. (k)-(m): © [2011] IEEE. Reprinted, with permission, from [11]. (n) from Middlebury Stereo Datasets(2005) [47] and © [2013] IEEE. Reprinted, with permission, from [48].
Fig. 7. Subjective comparisons on haze images. From left to right, the raw images and the results generated by Tarel et al. [42], He et al. [11], Zhu et al. [43], Berman et al. [44], Cai et al. [45], and the proposed method.
Fig. 8. Detailed comparison of real image dehazing. From left to right, the raw images and the results generated by Tarel et al. [42], He et al. [11], Zhu et al. [43], Berman et al. [44], Cai et al. [45], and the proposed method.
Further details are shown in Fig. 8 for a better display. Notably, the white coat and the sky region in the methods of He et al. [11] and Berman et al. [44] suffer severely from over-enhancement, as seen in Fig. 8(b) and Fig. 8(c), respectively. The former method does so because the sky region violates the theory of DCP, and the latter is over-enhanced due to the limitation of the prior. Moreover, it is seen that the position circled by the red rectangle of Cai et al.’s method [45] in Fig. 8(a) is darker due to the inaccuracy of their proposed CNN model, while other methods show more details in the same area. Zhu et al.’s method [43] has an incomplete dehazing effect, and the technique of Tarel [42] easily introduces noise. In contrast to these methods, the proposed method has a significant dehazing effect, and it barely produces an over-saturated region thanks to its accurate estimation of backscatter.
4.2 Subjective evaluation
For a fair and comprehensive evaluation, we select a variety of comparison methods, including model-free methods [24,26,27], model-based methods [15,16,21–23], and deep learning-based techniques [33,36], and carry out the contrast experiment on real underwater images with different color deviations and water depths. The real-world underwater images are from Sea-thru [9], UIEBD [36], and SQUID [23]. Part of the results is displayed in Fig. 9 and Fig. 10. To ensure the comparability of different approaches and to demonstrate the experimental results as best possible, the first five groups of images in Fig. 9 and Fig. 10 use the same original images (a-e), while the last five groups use completely different images (f-j).
Fig. 9. Subjective comparisons for real underwater images. (a) and (b) are images in greenish, (c) and (d) are images taken in coastal waters, (e)-(g) are images in turquoise, and (h)-(j) are images in bluish. From left to right, the raw images and the results generated by Ancuti et al. [24], Fu et al. [26], Li et al. [33], Li et al. [36], Lee et al. [27], and the proposed method.
Fig. 10. Subjective comparisons for real underwater images. (a) and (b) are images in greenish, (c) and (d) are images taken in coastal waters, (e) and (f) are images in turquoise, (g) and (h) are shallow-sea images in bluish, and (i) and (j) are deep-sea images in bluish. From left to right, the raw images and the results generated by Li et al. [21], Peng et al. [22], Berman et al. [23], Peng et al. [15], Zhou et al. [16], and the proposed method.
From Fig. 9, it is noted that the methods of Ancuti et al. [24] and Fu et al. [26] have the issue of amplifying noise. This is because the CLAHE algorithm included in both methods can well-enhance the edge details and easily introduce the noise. The CNN-based method of Li et al. [33] fails to reconstruct the actual color of underwater images when the appropriate water type is selected. The experimental results for the CNN-based method by Li et al. [36] show effectiveness, while further efforts are necessary to improve the clarity and details of images. This is partly because the model is trained with synthetic underwater images different from real underwater images. Although the approach of Lee et al. [27] produces a minimal reddish artifact, a large amount of noise is generated in the smooth background area due to the superpixel-based approach used to estimate the transmission map.
For the images shown in Fig. 10, it is evident that Li et al.’s approach [21] produces annoying red artifacts in most of the results (Fig. 10(b), (e)-(j)). The blurriness estimation algorithm of Peng et al. [22] fails to deal with greenish and bluish images (Fig. 10(a), (b), (i), (j)), and the problem of color distortion persists. This is because the method in [22] only considers eliminating the backscatter effects but fails to take color correction into account. In addition, it has low efficiency in estimating blurriness. Berman’s approach [23] results in low brightness and unnatural colors in most restored images, and Peng’s method [15] has unstable restoration results. Zhou et al. [16] achieve excellent subjective performance, but the step of reversing the red channel leads to a reddish color shift in some images (Fig. 10(a), (i), (j)), which makes it impossible to obtain the true colors of the underwater scene. In comparison, our method can handle various types of underwater images and obtain stable results with special enhanced effects.
To further test the practical application of our method, additional experiments are carried out on a real-world dataset, called RUIE (real-world underwater image enhancement, RUIE) [49]. It is a large underwater image dataset covering multiple real underwater images with various color casts, image qualities, and underwater scenes, mainly used to evaluate underwater image enhancement algorithms. Compared with several of the above-mentioned state-of-the-art methods, the experimental results on underwater images with different color casts and different qualities are shown in Fig. 11 and Fig. 12, respectively. As seen in Fig. 11, the methods of [15], [22], and [23] produce an unstable result, as these model-based methods barely consider color compensation, and the model of basis is not completely accurate. The fusion method of Ancuti et al. [24] obtains a good visual effect, which is much better than that of Fu et al. [26]. However, the results exhibit different degrees of redness in Fig. 11(e)-(h) due to the white balance algorithm used to discard unwanted color casts. In contrast, the proposed method effectively reduces color distortion without producing reddish artifacts. In Fig. 12, we note that images recovered by [15], [22], [23] and [26] from the high-quality category are usually clearer than those from the low-quality category. Although the methods of [24] obtain high contrast in each image, redness artifacts appear in Fig. 12(i), while the proposed method achieves both high contrast and natural colors.
Fig. 11. Results for underwater images with different color casts. (a)-(c) are greenish underwater images. (d)-(f) are bluish-green underwater images. (g)-(j) are blueish underwater images. From left to right, the raw images and the results generated by Ancuti et al. [24], Fu et al. [26], Peng et al. [22], Peng et al. [15], Berman et al. [23], and the proposed method.
Fig. 12. Results for underwater images with different quality. The image quality is divided into five categories from high to low with numbers from A to E, respectively. A represents the clearest quality, and E represents the most turbid quality. Among the ten images shown, (a), (b) belong to class A, (c), and (d) belong to class B, (e) and (f) belong to class C, (g), and (h) belong to class D and (i) and (j) belong to class E. From left to right, the raw images and the results generated by Ancuti et al. [24], Fu et al. [26], Peng et al. [22], Peng et al. [15], Berman et al. [23], and the proposed method.
According to further details shown in Fig. 13, the background color enhanced by [24] appears purple in Fig. 12(a), and the other results also feature different degrees of color distortion. By contrast, our method yields the most natural result. Figure 13(b) demonstrates that our approach has the most obvious dehazing effect. Moreover, the technique of Peng et al. [15] shows a significant decrease in clarity, while our method can well-restore the texture and details of the object in Fig. 13(c). In summary, our proposed method achieves superior results in color reconstruction, dehazing, and detail restoration.
Fig. 13. Detailed comparison results of different methods. (a), (b), (c) are three groups, and enhanced images in each group are from the same original images. From top to bottom, the raw images [9,36] and the results generated by Ancuti et al. [24], Peng et al. [22], Berman et al. [23], Peng et al. [15], and our method.
To further verify the accuracy of the proposed method in color reconstruction, we use a standard Macbeth color checker [50] as a reference and conduct a color accuracy experiment on underwater images with the same color chart. Results are displayed in Fig. 14. The recovered color chart of Ancuti’s method [24] and our method are closer to the standard color chart than other methods. Unfortunately, Ancuti’s method amplifies black noise in the overall enhanced image due to the CLAHE algorithm it uses to unveil details.
Fig. 14. Color accuracy test. (a), (b) and (c) are three underwater images that contain the standard Macbeth color checker taken by various waterproof cameras (captured by Pentax W80 (ISO 400), Panasonic TS1 (ISO 100) and Olympus Tough 8000 (ISO 100) from top to bottom, and the enhanced images in each group are from the same original images. For complete set please refer to the website Digital Photography Review (dpreview.com). We enlarge the color card of each result for observation, and the picture of standard Macbeth Color Checker . From left to right, the raw images and the results generated by Ancuti et al. [24], Peng et al. [22], Peng et al. [15], Berman et al. [23], and the proposed method.
4.3 Objective evaluation
The subjective results indicate that the proposed method has an outstanding performance in improving brightness, color, and contrast. Furthermore, it is comparable to the above-mentioned state-of-the-art approaches, and it even performs better in some ways. To verify the above subjective conclusions, we use objective evaluation methods, including UCIQE [51] and UIQM [52], to further assess the quality of the recovered images.
The UCIQE uses a linear combination of chroma, saturation, and contrast in CIELab space to quantify the quality of underwater images, and is expressed as follows:
where ${\sigma _c}$, $co{n_l}$, ${\mu _s}$ represents the standard deviation of chroma, the contrast of brightness, and the average of saturation, respectively. ${c_1}$, ${c_2}$ and ${c_3}$ define three weighted parameters were ${c_1}\textrm{ = 0}\textrm{.4680}$, ${c_2}\textrm{ = 0}\textrm{.2745}$ and ${c_3}\textrm{ = 0}\textrm{.2576}$. The value of UCIQE is between 0 and 1. The greater the value, the higher the image quality.The UIQM evaluates the quality of underwater images through a linear combination of sharpness (UISM), colorfulness (UICM), and contrast (UIConM), and is expressed as follows:
where ${c_1}$, ${c_2}$ and ${c_3}$ indicates the weighted factors, and the obtained coefficients are ${c_1}\textrm{ = 0}\textrm{.0282}$, ${c_2}\textrm{ = 0}\textrm{.2953}$, ${c_3}\textrm{ = 3}\textrm{.5753}$. The best quality of an underwater image can obtain the highest value of UIQM.Since Fig. 9 and Fig. 10 contain a variety of methods, and part of the results come from the same original images, we subsequently evaluate the quality of recovered images in Fig. 9 and Fig. 10 through UIQM and UCIQE, and the evaluation values are respectively displayed in Fig. 15 and Fig. 16. It can be seen that the method proposed by Li et al. [21] has the highest objective evaluation value. Unfortunately, in combination with the subjective evaluation results, it produces red artifacts and unnatural colors, which cause the high value of objective evaluation. In other words, brighter colors and higher brightness are the reasons for its higher objective evaluation value, which, in some cases, demonstrates the limitations of the current underwater image quality measures. On the other hand, the subjective result of Ancuti et al. [24] is relatively better, but its objective value is lower than that of the proposed method in most cases. In summary, compared to the current state-of-the-art approaches, the proposed method has a relatively high objective evaluation value and obtains higher contrast and more natural details without producing noise, artifact, and overexposure.
Fig. 15. UCIQE (a) and UIQM (b) values in Fig. 9. The horizontal axis represents each method. From left to right, the methods of Ancuti et al. [24], Fu et al. [26], Li et al. [33], Li et al. [36], Lee et al. [27], and our method. The vertical axis represents the evaluation values of images from (a) to (j) in Fig. 9. The red line represents the average of all image evaluation values for each method.
Fig. 16. UCIQE (a) and UIQM (b) values in Fig. 10. The horizontal axis represents each method. From left to right, the methods of Li et al. [21], Peng et al. [22], Berman et al. [23], Peng et al. [15], Zhou et al. [16] and the proposed method. The vertical axis represents the evaluation values of images from (a) to (j) in Fig. 10. The red line represents the average of all image evaluation values of each method.
5. Discussion
Research on underwater image enhancement undoubtedly plays a crucial role in ocean exploration [1–3]. However, most model-based methods remain unstable when coping with underwater images taken in various water types. The image formation model (IFM) applied in these methods is one of the underlying reasons. Herein, a revised model was proposed that takes more factors into account, including a depth map, and this model showed better performance and stability for correcting complex underwater scenes. Unfortunately, current restoration methods barely opt to take the revised model as a base due to its complexity. Besides, methods of depth map estimation for underwater images are lacking. Considering the problems mentioned above, we focused on calculating backscatter depending on the imaging range estimated by a novel approach while ignoring the influence of other parameters. Notwithstanding, several experiments showed that images restored by the proposed method are much better than results based on IFM. In addition, we established that color distortion still exists after removing the backscatter. Therefore, a color correction method was presented to reduce the problem effectively. Both color accuracy tests and experiments on different color casts demonstrated that the proposed method outperforms previous methods in almost all cases. Moreover, our approach shows outstanding performance when applied to real-world underwater images.
6. Conclusions
In this study, we proposed an underwater image enhancement approach that includes two processes. Firstly, in underwater image dehazing, a novel method was designed to estimate the imaging range. The backscatter was then removed with the depth map based on a model with enhanced accuracy. Secondly, in the process of color cast elimination, we proposed an approach to estimate the best illumination parameters in each channel of the dehazing underwater image. To verify the effectiveness of the proposed method, we divided the experiment into two parts, namely a dehazing contrast experiment and an underwater image enhancement contrast experiment from both objective and subjective aspects. All results proved that our method can obtain high-quality underwater images.
Despite the improved results for processing various underwater images compared with current state-of-the-art techniques, the accuracy of the proposed imaging range estimation method still has its limitation: the overestimation or underestimation of backscatter. Furthermore, the existing segmentation methods and depth map estimation methods for underwater images are still scarce. Thus, we will attempt to estimate the imaging range of underwater images more accurately in future work.
Funding
Fundamental Research Funds for the Central Universities (3132019205, 3132019354); Liaoning Provincial Natural Science Foundation of China (20170520196); National Natural Science Foundation of China (61702074).
Acknowledgment
The authors acknowledge the financial funding of this work. We also thank the anonymous reviewers for their critical comments on the manuscript.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. K. Liu and Y. Liang, “Underwater image enhancement method based on adaptive attenuation-curve prior,” Opt. Express 29(7), 10321–10345 (2021). [CrossRef]
2. M. Ludvigsen, B. Sortland, G. Johnsen, and S. Hanumant, “Applications of geo-referenced underwater photo mosaics in marine biology and archaeology,” J. Oceanog. 20(4), 140–149 (2007). [CrossRef]
3. C. Alippi, R. Camplani, C. Galperti, and M. Roveri, “A robust, adaptive, solar-powered WSN framework for aquatic environmental monitoring,” IEEE Sensors J. 11(1), 45–55 (2011). [CrossRef]
4. Z. Duan, Y. Yuan, J. C. Lu, J. L. Wang, Y. Li, S. Svanberg, and G. Y. Zhao, “Underwater spatially, spectrally, and temporally resolved optical monitoring of aquatic fauna,” Opt. Express 28(2), 2600–2610 (2020). [CrossRef]
5. T. Cui, Q. Song, J. Tang, and J. Zhang, “Spectral variability of sea surface skylight reflectance and its effect on ocean color,” Opt. Express 21(21), 24929–24941 (2013). [CrossRef]
6. J. Wei, Z. P. Lee, M. Lewis, N. Pahlevan, M. Ondrusek, and R. Armstrong, “Radiance transmittance measured at the ocean surface,” Opt. Express 23(9), 11826–11837 (2015). [CrossRef]
7. D. Akkaynak, T. Treibitz, T. Shlesinger, R. Tamir, Y. Loya, and D. Iluz, “What is the space of attenuation coefficients in underwater computer vision?” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), (2017), pp. 568–577.
8. D. Akkaynak and T. Treibitz, “A revised underwater image formation model,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), (2018), pp. 6723–6732.
9. D. Akkaynak and T. Treibitz, “Sea-thru: A method for removing water from underwater images,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), (2019), pp. 1682–1691.
10. H. Koschmieder, “Theorie der horizontalen Sichtweite. Beitrage zur Physik der freien Atmosphare,” Meteorol. Z 12, 3353 (1924).
11. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2011). [CrossRef]
12. C. John Y and C. Ying-Ching, “Underwater image enhancement by wavelength compensation and dehazing,” IEEE Trans. Image Process. 21(4), 1756–1769 (2012). [CrossRef]
13. P. Drews, E. Nascimento, F. Moraes, S. Botelho, and M. Campos, “Transmission estimation in underwater single images,” in Proc. IEEE International Conference on Computer Vision Workshops (ICCVW) (2013), pp. 825–830.
14. A. Galdran, D. Pardo, A. Picón, and A. Alvarez-Gila, “Automatic red-channel underwater image restoration,” J. Vis. Commun. Image Represent. 26, 132–145 (2015). [CrossRef]
15. Y. Peng, K. Cao, and P. C. Cosman, “Generalization of the Dark Channel Prior for Single Image Restoration,” IEEE Trans. Image Process. 27(6), 2856–2868 (2018). [CrossRef]
16. J. Zhou, Z. Liu, W. Zhang, D. Zhang, and W. Zhang, “Underwater image restoration based on secondary guided transmission map,” Multimed. Tools Appl. 80(5), 7771–7788 (2021). [CrossRef]
17. L. Chao and M. Wang, “Removal of water scattering,” in Proc. IEEE 2nd Int. Conf. Comput. Eng. Technol (CCET), (2010), pp. 35–39.
18. H.-Y. Yang, P.-Y. Chen, C.-C. Huang, Y.-Z. Zhuang, and Y.-H. Shiau, “Low complexity underwater image enhancement based on dark channel prior,” in Proc. IEEE 2nd Int. Conf. Innov. Bio-inspired Comput. Appl (IBICA), (Dec. 2011), pp. 17–20.
19. H. Wen, Y. Tian, T. Huang, and W. Gao, “Single underwater image enhancement with a new optical model,” in: Proc. IEEE Int. Symposium on Circuits and Systems (ISCAS), (2013), pp. 753–756.
20. N. Carlevaris-Bianco, A. Mohan, and R. M. Eustice, “Initial results in underwater single image dehazing,” in Proc. OCEANS MTS/IEEE SEATTLE (2010), pp. 1–8.
21. C. Li, J. Guo, R. Cong, Y. Pang, and B. Wang, “Underwater Image Enhancement by Dehazing With Minimum Information Loss and Histogram Distribution Prior,” IEEE Trans. Image Process. 25(12), 5664–5677 (2016). [CrossRef]
22. Y. Peng and P. C. Cosman, “Underwater Image Restoration Based on Image Blurriness and Light Absorption,” IEEE Trans. Image Process. 26(4), 1579–1594 (2017). [CrossRef]
23. D. Berman, D. Levy, S. Avidan, and T. Treibitz. “Underwater Single Image Color Restoration Using Haze-Lines and a New Quantitative Dataset,” IEEE Trans. Pattern Anal. Mach. Intell. (2020). [CrossRef]
24. C. Ancuti, C.O. Ancuti, T. Haber, and P. Bekaert, “Enhancing underwater images and videos by fusion,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), (2012), pp. 81–88.
25. C. O. Ancuti, C. Ancuti, D. Vleeschouwer, and P. Bekaert, “Color balance and fusion for underwater image enhancement,” IEEE Trans. Image Process. 27(1), 379–393 (2018). [CrossRef]
26. X. Fu, Z. Fan, M. Ling, Y. Huang, and X. Ding, “Two-step approach for single underwater image enhancement,” in Proc. Int. Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), (2017), pp. 789–794.
27. H. S. Lee, S. W. Moon, and I. K. Eom, “Underwater image enhancement using successive color correction and superpixel dark channel prior,” Symmentry 12(8), 1220 (2020). [CrossRef]
28. K. He, J. Sun, and X. Tang, “Guided Image Filtering,” IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013). [CrossRef]
29. C. Guo, C. Li, J. Guo, R. Cong, H. Fu, and P. Han, “Hierarchical features driven residual learning for depth map super-resolution,” IEEE Trans. Image Process. 28(5), 2545–2557 (2019). [CrossRef]
30. K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning Deep CNN Denoiser Prior for Image Restoration,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), (2017), pp. 2808–2817.
31. Y. Guo, H. Li, and P. Zhuang, “Underwater image enhancement using a multiscale dense generative adversarial network,” IEEE J. Ocean. Eng. 45(3), 862–870 (2020). [CrossRef]
32. J. Lu, N. Li, S. Zhang, Z. Yu, H. Zheng, and B. Zheng, “Multi-scale adversarial network for underwater image restoration,” Opt. Laser Technol. 110, 105–113 (2019). [CrossRef]
33. C. Li, S. Anwar, and F. Porikli, “Underwater scene prior inspired deep underwater image and video enhancement,” Pattern Recognit. 98, 107038 (2020). [CrossRef]
34. J. Li, K. A. Skinner, R. M. Eustice, and M. Johnson-Roberson, “WaterGAN: Unsupervised Generative Network to Enable Real-time Color Correction of Monocular Underwater Images,” IEEE Robot. Autom. Lett. 3(1), 1 (2017). [CrossRef]
35. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in 27th Int. Conf. on Neural Information Processing Systems (NIPS) (2014), pp. 2672–2680.
36. C. Li, C. Guo, W. Ren, R. Cong, J. Hou, S. Kwong, and D. Tao, “An Underwater Image Enhancement Benchmark Dataset and Beyond,” IEEE Trans. Image Process. 29, 4376–4389 (2020). [CrossRef]
37. A. Saxena, S. H. Chung, and A. Y. Ng, “Learning depth from single monocular images,” in 18th Int. Conf. on Neural Information Processing Systems (NIPS) (2005), pp. 1161–1168.
38. A. Saxena, J. Schulte, and A. Y. Ng, “Depth estimation using monocular and stereo cues,” in 20th Int. Joint Conf. on Artificial Intelligence (IJCAL) (2007), pp. 2197–2203.
39. M. Liu, M. Salzmann, and X. He, “Discrete-continuous depth estimation from a single image,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), (2014), pp. 716–723.
40. C. Godard, O. Mac Aodha, M. Firman, and G. Brostow, “Digging into self-supervised monocular depth estimation,” in Proc. IEEE/CVF Int. Conf. on Computer Vision (ICCV), (2019), pp. 3827–3837.
41. R. C. Gonzalez, R. E. Woods, and S. L. Eddins, Digital Image Process Using MATLAB, 3rd ed (Prentice-Hall, Upper Saddle River, 2008).
42. J. Tarel and N. Hautière, “Fast visibility restoration from a single color or gray level image,” in IEEE 12th Int. Conf. on Computer Vision (ICCV), (2009), pp. 2201–2208.
43. Q. Zhu, J. Mai, and L. Shao, “A Fast Single Image Haze Removal Algorithm Using Color Attenuation Prior,” IEEE Trans. Image Process. 24(11), 3522–3533 (2015). [CrossRef]
44. D. Berman, T. Treibitz, and S. Avidan, “Non-local Image Dehazing,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), (2016), pp. 1674–1682.
45. B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, “DehazeNet: An End-to-End System for Single Image Haze Removal,” IEEE Trans. Image Process. 25(11), 5187–5198 (2016). [CrossRef]
46. B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, “Benchmarking Single-Image Dehazing and Beyond,” IEEE Trans. Image Process. 28(1), 492–505 (2019). [CrossRef]
47. D. Scharstein and C. Pal, “Learning conditional random fields for stereo,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), (2007), pp. 1–8.
48. C. O. Ancuti and C. Ancuti, “Single image dehazing by multi-scale fusion,” IEEE Trans. Image Process. 22(8), 3271–3282 (2013). [CrossRef]
49. R. Liu, X. Fan, M. Zhu, M. Hou, and Z. Luo, “Real-World Underwater Enhancement: Challenges, Benchmarks, and Solutions Under Natural Light,” IEEE Trans. Circuits Syst. Video Technol. 30(12), 4861–4875 (2020). [CrossRef]
50. R. Kreslin, P. M. Calvo, L. G. Corzo, and P. Peer, “Linear Chromatic Adaptation Transform based on Delaunay Triangulation,” Math. Probl. Eng. 2014(760123), 1–9 (2014). [CrossRef]
51. M. Yang and A. Sowmya, “An underwater color image quality evaluation metric,” IEEE Trans. Image Process. 24(12), 6062–6071 (2015). [CrossRef]
52. K. Panetta, C. Gao, and S. Agaian, “Human-visual-system-inspired underwater image quality measures,” IEEE J. Ocean. Eng. 41(3), 541–551 (2016). [CrossRef]
